Nov 03 20:44:35.946275 ip-10-0-130-134 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 20:44:35.946287 ip-10-0-130-134 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 20:44:35.946297 ip-10-0-130-134 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 20:44:35.946607 ip-10-0-130-134 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 20:44:46.195408 ip-10-0-130-134 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 20:44:46.195425 ip-10-0-130-134 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot f4d3bae6cb224a2baa622c6d6f9110bf -- Nov 03 20:46:47.271322 ip-10-0-130-134 systemd[1]: Starting Kubernetes Kubelet... Nov 03 20:46:47.674692 ip-10-0-130-134 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:46:47.674692 ip-10-0-130-134 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 20:46:47.674692 ip-10-0-130-134 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:46:47.674692 ip-10-0-130-134 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:46:47.678647 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678547 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:46:47.678647 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678642 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:46:47.678647 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678647 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:46:47.678647 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678651 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:46:47.678647 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678653 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:46:47.678647 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678657 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678660 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678663 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678666 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678668 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678671 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678674 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678676 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678679 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678682 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678684 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678687 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678690 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678692 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678695 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678697 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678700 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678703 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678706 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678708 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:46:47.678871 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678711 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678713 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678716 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678721 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678724 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678727 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678730 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678732 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678735 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678737 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678740 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678743 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678745 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678748 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678752 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678754 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678757 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678760 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678763 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678766 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:46:47.679346 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678768 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678771 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678774 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678776 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678779 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678786 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678788 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678791 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678794 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678796 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678799 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678802 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678804 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678807 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678809 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678812 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678814 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678817 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678819 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678822 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:46:47.679806 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678824 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678827 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678829 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678834 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678839 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678842 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678846 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678848 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678852 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678854 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678857 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678859 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678864 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678868 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678872 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678875 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678879 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678881 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678885 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:46:47.680292 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678888 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678890 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678893 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678895 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678898 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678913 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678916 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678919 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678922 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678924 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678927 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678930 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678933 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678935 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678938 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678941 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678943 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678946 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678949 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678952 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:46:47.680780 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678955 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.678959 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679773 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679780 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679783 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679786 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679789 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679792 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679795 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679797 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679800 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679803 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679806 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679809 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679812 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679815 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679817 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679820 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679822 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679825 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:46:47.681863 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679827 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679830 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679832 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679835 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679838 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679840 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679842 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679845 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679848 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679850 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679853 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679856 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679859 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679862 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679864 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679866 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679870 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679872 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679875 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679877 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:46:47.682365 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679880 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679883 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679885 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679888 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679892 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679897 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679900 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679917 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679921 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679924 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679927 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679930 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679932 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679935 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679937 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679940 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679942 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679945 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679948 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:46:47.682842 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679950 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679953 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679955 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679959 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679964 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679968 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679972 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679975 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679978 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679981 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679984 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679986 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679989 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679991 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679994 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.679997 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680000 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680002 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680005 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:46:47.683299 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680008 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680010 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680013 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680016 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680018 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680021 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680023 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680028 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680030 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680033 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680035 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680038 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680040 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680043 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680045 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680048 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680051 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680054 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680057 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:46:47.683745 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680059 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680062 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680065 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680067 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680070 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680072 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680075 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680077 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680080 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680083 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680086 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680623 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680630 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680634 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680638 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680640 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680644 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680647 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680650 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680653 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:46:47.684199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680656 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680658 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680661 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680663 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680666 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680668 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680671 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680674 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680677 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680680 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680683 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680686 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680688 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680691 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680693 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680696 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680698 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680700 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680703 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680706 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:46:47.684673 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680708 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680711 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680713 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680715 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680718 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680720 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680725 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680729 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680732 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680736 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680738 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680750 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680753 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680756 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680759 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680761 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680764 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680767 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680770 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:46:47.685235 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680773 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680775 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680778 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680785 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680788 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680790 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680793 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680796 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680799 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680801 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680804 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680806 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680809 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680813 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680816 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680818 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680821 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680824 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680826 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680828 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:46:47.685694 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680837 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680840 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680842 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680845 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680847 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680850 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680852 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680855 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680857 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680860 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680862 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680865 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680867 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680870 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680873 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680876 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680879 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680882 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680884 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:46:47.686219 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680886 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680889 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680891 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680894 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680897 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680899 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680917 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680919 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680922 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680925 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680927 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680930 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680934 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680936 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680939 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680942 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680945 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680947 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.680950 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681043 2583 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 20:46:47.686665 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681066 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681078 2583 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681082 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681087 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681090 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681095 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681099 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681102 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681106 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681110 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681113 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681116 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681120 2583 flags.go:64] FLAG: --cgroup-root="" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681123 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681126 2583 flags.go:64] FLAG: --client-ca-file="" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681129 2583 flags.go:64] FLAG: --cloud-provider="external" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681131 2583 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681138 2583 flags.go:64] FLAG: --cluster-domain="" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681143 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681146 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681149 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681152 2583 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681156 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 20:46:47.687175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681159 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681162 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681166 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681169 2583 flags.go:64] FLAG: --contention-profiling="false" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681178 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681181 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681184 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681187 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681192 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681195 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681198 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681201 2583 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681204 2583 flags.go:64] FLAG: --enable-server="true" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681207 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681213 2583 flags.go:64] FLAG: --event-burst="100" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681216 2583 flags.go:64] FLAG: --event-qps="50" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681219 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681223 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681226 2583 flags.go:64] FLAG: --eviction-hard="" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681231 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681234 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681237 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681239 2583 flags.go:64] FLAG: --eviction-soft="" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681243 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681245 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 20:46:47.687705 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681248 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681251 2583 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681256 2583 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681259 2583 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681261 2583 flags.go:64] FLAG: --feature-gates="" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681266 2583 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681269 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681272 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681276 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681279 2583 flags.go:64] FLAG: --healthz-port="10248" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681282 2583 flags.go:64] FLAG: --help="false" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681285 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681289 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681293 2583 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681295 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681299 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681302 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681305 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681308 2583 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681311 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681314 2583 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681317 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681320 2583 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681323 2583 flags.go:64] FLAG: --kube-reserved="" Nov 03 20:46:47.688342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681327 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681330 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681333 2583 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681336 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681339 2583 flags.go:64] FLAG: --lock-file="" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681342 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681345 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681348 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681354 2583 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681357 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681361 2583 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681364 2583 flags.go:64] FLAG: --logging-format="text" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681367 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681371 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681373 2583 flags.go:64] FLAG: --manifest-url="" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681377 2583 flags.go:64] FLAG: --manifest-url-header="" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681381 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681385 2583 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681389 2583 flags.go:64] FLAG: --max-pods="110" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681392 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681395 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681399 2583 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681402 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681405 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681408 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 20:46:47.688965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681411 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681419 2583 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681422 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681425 2583 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681428 2583 flags.go:64] FLAG: --pod-cidr="" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681431 2583 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681434 2583 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681439 2583 flags.go:64] FLAG: --pods-per-core="0" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681442 2583 flags.go:64] FLAG: --port="10250" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681445 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681448 2583 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0f708d82c0704dc34" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681451 2583 flags.go:64] FLAG: --qos-reserved="" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681454 2583 flags.go:64] FLAG: --read-only-port="10255" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681457 2583 flags.go:64] FLAG: --register-node="true" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681460 2583 flags.go:64] FLAG: --register-with-taints="" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681464 2583 flags.go:64] FLAG: --registry-burst="10" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681466 2583 flags.go:64] FLAG: --registry-qps="5" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681472 2583 flags.go:64] FLAG: --reserved-cpus="" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681475 2583 flags.go:64] FLAG: --reserved-memory="" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681478 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681482 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681485 2583 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681488 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681491 2583 flags.go:64] FLAG: --runonce="false" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681494 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681497 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681500 2583 flags.go:64] FLAG: --seccomp-default="false" Nov 03 20:46:47.689590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681503 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681506 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681510 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681513 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681516 2583 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681519 2583 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681522 2583 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681525 2583 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681528 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681531 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681534 2583 flags.go:64] FLAG: --system-cgroups="" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681537 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681543 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681546 2583 flags.go:64] FLAG: --tls-cert-file="" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681549 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681553 2583 flags.go:64] FLAG: --tls-min-version="" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681557 2583 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681560 2583 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681563 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681565 2583 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681568 2583 flags.go:64] FLAG: --v="2" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681574 2583 flags.go:64] FLAG: --version="false" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681580 2583 flags.go:64] FLAG: --vmodule="" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681584 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.681587 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 20:46:47.690250 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681740 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681744 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681747 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681750 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681752 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681755 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681758 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681760 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681762 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681765 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681768 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681771 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681774 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681776 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681779 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681782 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681785 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681789 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681792 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681796 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:46:47.690847 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681799 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681801 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681804 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681806 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681809 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681811 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681814 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681816 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681819 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681823 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681826 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681829 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681831 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681834 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681837 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681839 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681842 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681845 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681849 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681852 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:46:47.691331 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681855 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681858 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681860 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681864 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681866 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681869 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681872 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681874 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681877 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681880 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681883 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681886 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681889 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681892 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681894 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681897 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681899 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681914 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681917 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681920 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:46:47.691819 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681922 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681927 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681929 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681932 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681935 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681937 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681940 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681942 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681945 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681947 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681950 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681953 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681955 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681958 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681961 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681963 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681966 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681969 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681971 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:46:47.692324 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681974 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681976 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681979 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681982 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681985 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681988 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681990 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681993 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681995 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.681998 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682001 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682003 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682006 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682009 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682012 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682015 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682018 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682021 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682023 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682026 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:46:47.692787 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682029 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:46:47.693342 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682031 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:46:47.693342 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682034 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:46:47.693342 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682037 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:46:47.693342 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682039 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:46:47.693342 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682042 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:46:47.693342 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682044 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:46:47.693342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.682070 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:46:47.693342 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682106 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:46:47.693342 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682110 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:46:47.693342 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682113 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:46:47.693342 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682117 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:46:47.693342 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682120 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:46:47.693342 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682122 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:46:47.693342 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682125 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:46:47.693342 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682127 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:46:47.693342 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682131 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682134 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682136 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682139 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682142 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682145 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682147 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682150 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682153 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682155 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682158 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682163 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682166 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682168 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682171 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682174 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682176 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682179 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682181 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682184 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:46:47.693719 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682186 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682189 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682192 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682194 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682198 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682202 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682205 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682208 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682210 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682213 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682215 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682218 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682222 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682225 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682227 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682229 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682232 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682235 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682237 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682240 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:46:47.694199 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682242 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682245 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682247 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682251 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682253 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682256 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682260 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682263 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682267 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682269 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682273 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682275 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682278 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682280 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682283 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682285 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682288 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682290 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682293 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682295 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:46:47.694665 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682298 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682301 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682303 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682306 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682310 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682313 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682315 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682318 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682320 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682323 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682326 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682329 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682331 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682334 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682337 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682340 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682343 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682345 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682348 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:46:47.695176 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682351 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682353 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682356 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682359 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682361 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682364 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682367 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682369 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682372 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682375 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682377 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682381 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682384 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682386 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682390 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682393 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682395 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682399 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.682402 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:46:47.695636 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.688888 2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.688921 2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.690262 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.690269 2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690337 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690342 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690346 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690349 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690352 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690354 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690357 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690360 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690362 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690365 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690367 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690370 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690373 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690375 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690378 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690381 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:46:47.696091 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690384 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:46:47.696573 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690386 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:46:47.696573 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690390 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:46:47.696573 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690394 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:46:47.696573 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690397 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:46:47.696573 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690400 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:46:47.696573 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690403 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:46:47.696573 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690406 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:46:47.696573 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690408 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:46:47.696573 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690410 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:46:47.696573 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690413 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:46:47.696573 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690416 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:46:47.696573 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690419 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:46:47.696573 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690428 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:46:47.696573 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690431 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:46:47.696573 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690433 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:46:47.696573 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690436 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:46:47.696573 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690439 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:46:47.696573 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690443 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690446 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690449 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690453 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690456 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690459 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690462 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690464 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690467 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690469 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690472 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690474 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690477 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690480 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690482 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690485 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690487 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690490 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690492 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690495 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690498 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:46:47.697065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690500 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690503 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690505 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690508 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690510 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690513 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690516 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690525 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690528 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690531 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690533 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690536 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690538 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690541 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690543 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690546 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690549 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690551 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690554 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690556 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:46:47.697553 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690558 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690561 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690564 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690566 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690569 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690571 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690574 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690576 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690579 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690581 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690584 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690586 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690589 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690592 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690594 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690596 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690599 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690601 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690605 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:46:47.698316 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690607 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:46:47.699092 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690615 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:46:47.699092 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690618 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:46:47.699092 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690621 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:46:47.699092 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690623 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:46:47.699092 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690626 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:46:47.699092 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690628 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:46:47.699092 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690631 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:46:47.699092 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690634 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:46:47.699092 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690637 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:46:47.699092 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690639 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:46:47.699092 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690642 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:46:47.699092 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.690671 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:46:47.699092 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690855 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:46:47.699092 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690859 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:46:47.699092 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690862 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690865 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690868 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690870 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690873 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690876 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690878 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690881 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690884 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690886 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690889 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690891 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690894 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690897 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690899 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690914 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690919 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690924 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690927 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690936 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:46:47.699457 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690939 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690942 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690945 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690947 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690950 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690952 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690955 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690958 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690961 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690964 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690967 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690970 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690972 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690975 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690978 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690980 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690983 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690985 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690988 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:46:47.700009 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690990 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690993 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.690997 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691000 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691003 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691006 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691009 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691011 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691014 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691017 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691020 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691023 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691026 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691033 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691036 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691039 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691042 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691044 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691046 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:46:47.700531 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691049 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691052 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691054 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691057 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691059 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691061 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691064 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691067 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691069 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691072 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691074 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691077 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691079 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691082 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691084 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691094 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691096 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691099 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691101 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:46:47.701161 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691104 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691106 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691109 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691112 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691114 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691117 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691120 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691122 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691131 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691134 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691137 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691139 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691142 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691144 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691147 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691149 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691152 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691154 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691157 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691159 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:46:47.701670 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691162 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:46:47.702394 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691165 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:46:47.702394 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691167 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:46:47.702394 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691170 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:46:47.702394 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691172 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:46:47.702394 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691175 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:46:47.702394 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691177 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:46:47.702394 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.691201 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:46:47.702394 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691238 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:46:47.702394 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691242 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:46:47.702394 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691245 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:46:47.702394 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691248 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:46:47.702394 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691250 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:46:47.702394 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691253 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:46:47.702394 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691256 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:46:47.702394 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691259 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:46:47.702394 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691261 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691264 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691267 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691269 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691272 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691280 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691283 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691285 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691288 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691291 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691294 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691296 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691298 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691301 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691303 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691306 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691308 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691311 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691313 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691316 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:46:47.702808 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691318 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691320 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691323 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691325 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691328 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691330 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691333 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691336 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691338 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691341 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691344 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691346 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691349 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691352 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691355 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691357 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691360 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691362 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691370 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691373 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:46:47.703293 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691376 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691378 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691380 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691383 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691385 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691388 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691390 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691395 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691398 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691402 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691405 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691407 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691410 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691412 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691415 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691417 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691420 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691422 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691425 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:46:47.703761 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691427 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691430 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691434 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691437 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691440 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691443 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691446 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691448 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691451 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691454 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691456 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691459 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691467 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691470 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691472 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691475 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691477 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691480 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691482 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:46:47.704213 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691485 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691488 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691490 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691493 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691495 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691498 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691500 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691503 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691505 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691508 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691510 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691513 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691516 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691519 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691521 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691524 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691526 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691531 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691534 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:46:47.704663 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:47.691536 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:46:47.705186 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.691598 2583 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 20:46:47.705186 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.693637 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 20:46:47.705186 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.694427 2583 server.go:1014] "Starting client certificate rotation" Nov 03 20:46:47.705186 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.694531 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:46:47.705186 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.694572 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:46:47.705186 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.696343 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:46:47.705186 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.700389 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:46:47.712393 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.712376 2583 log.go:25] "Validated CRI v1 runtime API" Nov 03 20:46:47.716298 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.716283 2583 log.go:25] "Validated CRI v1 image API" Nov 03 20:46:47.717155 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.717129 2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 20:46:47.719858 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.719830 2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c79ab3fe-ba33-4c36-b0d9-d0963013f182:/dev/nvme0n1p4 e463d441-011b-4dd7-9143-60ee9243974a:/dev/nvme0n1p3] Nov 03 20:46:47.719932 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.719858 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 20:46:47.724964 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.724841 2583 manager.go:217] Machine: {Timestamp:2026-11-03 20:46:47.723523307 +0000 UTC m=+0.324516342 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101064 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25084211570d2077fa2d96f8dd88a2 SystemUUID:ec250842-1157-0d20-77fa-2d96f8dd88a2 BootID:f4d3bae6-cb22-4a2b-aa62-2c6d6f9110bf Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:c2:30:11:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:c2:30:11:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:91:ff:5c:68:09 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 20:46:47.724964 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.724963 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 20:46:47.725093 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.725081 2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 20:46:47.728245 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.728219 2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 20:46:47.728399 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.728246 2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-134.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 20:46:47.728449 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.728402 2583 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 20:46:47.728449 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.728410 2583 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 20:46:47.728498 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.728459 2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:46:47.728498 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.728478 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:46:47.729259 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.729249 2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 20:46:47.732965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.732953 2583 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 20:46:47.733082 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.733073 2583 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 20:46:47.733761 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.733749 2583 kubelet.go:493] "Attempting to sync node with API server" Nov 03 20:46:47.733808 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.733765 2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 20:46:47.733808 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.733775 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 20:46:47.733808 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.733785 2583 kubelet.go:405] "Adding apiserver pod source" Nov 03 20:46:47.733808 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.733794 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 20:46:47.734142 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.734126 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:46:47.736559 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.736534 2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 20:46:47.741362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.741343 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:46:47.742372 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.742358 2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 20:46:47.742441 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.742398 2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 20:46:47.742624 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.742608 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 20:46:47.742683 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.742639 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 20:46:47.742683 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.742668 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 20:46:47.742683 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.742677 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 20:46:47.742773 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.742687 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 20:46:47.742773 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.742696 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 20:46:47.743236 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.743225 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 20:46:47.743295 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.743238 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 20:46:47.743295 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.743251 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 20:46:47.743295 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.743261 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 20:46:47.743295 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.743280 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 20:46:47.743295 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.743293 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 20:46:47.744371 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.744359 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 20:46:47.744425 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.744374 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 20:46:47.748427 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.748411 2583 server.go:1263] "Started kubelet" Nov 03 20:46:47.748527 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.748477 2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 20:46:47.748636 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.748579 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 20:46:47.752212 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.752170 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 20:46:47.752754 ip-10-0-130-134 systemd[1]: Started Kubernetes Kubelet. Nov 03 20:46:47.754765 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.753941 2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 20:46:47.756130 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.756113 2583 server.go:317] "Adding debug handlers to kubelet server" Nov 03 20:46:47.758341 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:47.758321 2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 20:46:47.761430 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.761407 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 20:46:47.761846 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.761826 2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 20:46:47.761973 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:47.761947 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-134.us-west-2.compute.internal\" not found" Nov 03 20:46:47.762438 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.762421 2583 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 20:46:47.762438 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.762440 2583 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 20:46:47.762551 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.762419 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 20:46:47.762551 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.762525 2583 reconstruct.go:97] "Volume reconstruction finished" Nov 03 20:46:47.762551 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.762534 2583 reconciler.go:29] "Reconciler: start to sync state" Nov 03 20:46:47.762799 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.762782 2583 factory.go:55] Registering systemd factory Nov 03 20:46:47.762799 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.762798 2583 factory.go:223] Registration of the systemd container factory successfully Nov 03 20:46:47.763041 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.763024 2583 factory.go:153] Registering CRI-O factory Nov 03 20:46:47.763041 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.763043 2583 factory.go:223] Registration of the crio container factory successfully Nov 03 20:46:47.763156 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.763146 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 20:46:47.763189 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.763165 2583 factory.go:103] Registering Raw factory Nov 03 20:46:47.763189 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.763176 2583 manager.go:1199] Started watching for new ooms in manager Nov 03 20:46:47.764073 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.764051 2583 manager.go:319] Starting recovery of all containers Nov 03 20:46:47.770550 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.770502 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 20:46:47.776691 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.776669 2583 manager.go:324] Recovery completed Nov 03 20:46:47.783882 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.783865 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:46:47.786165 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.786148 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:46:47.786245 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.786185 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:46:47.786245 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.786198 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:46:47.786762 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.786751 2583 cpu_manager.go:226] "Starting" policy="none" Nov 03 20:46:47.786762 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.786759 2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 20:46:47.786872 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.786764 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:46:47.786872 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.786776 2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 20:46:47.787575 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.787561 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:46:47.787633 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.787588 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:46:47.787633 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.787597 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:46:47.789045 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.789031 2583 policy_none.go:50] "Start" Nov 03 20:46:47.789112 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.789050 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 20:46:47.789112 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.789066 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 20:46:47.791077 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.791065 2583 policy_none.go:44] "Start" Nov 03 20:46:47.831616 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.831568 2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 20:46:47.835666 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.835652 2583 manager.go:341] "Starting Device Plugin manager" Nov 03 20:46:47.835734 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:47.835687 2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 20:46:47.835734 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.835697 2583 server.go:87] "Starting device plugin registration server" Nov 03 20:46:47.835983 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.835969 2583 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 20:46:47.836056 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.835983 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 20:46:47.836116 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.836077 2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 20:46:47.836289 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.836207 2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 20:46:47.836289 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.836227 2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 20:46:47.836798 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:47.836779 2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 20:46:47.836872 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:47.836825 2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-134.us-west-2.compute.internal\" not found" Nov 03 20:46:47.894167 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.893809 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-134.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 20:46:47.894167 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:47.893830 2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-134.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 20:46:47.895946 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:47.893801 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-134.us-west-2.compute.internal.18e4a3f127acd6a1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-134.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-134.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:46:47.748384417 +0000 UTC m=+0.349377452,LastTimestamp:2026-11-03 20:46:47.748384417 +0000 UTC m=+0.349377452,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-134.us-west-2.compute.internal,}" Nov 03 20:46:47.899429 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.899403 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 20:46:47.899429 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.899431 2583 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 20:46:47.899564 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.899449 2583 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 20:46:47.899564 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:47.899479 2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 20:46:47.936465 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.936383 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:46:47.938828 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.938808 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:46:47.938956 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.938842 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:46:47.938956 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.938853 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:46:47.938956 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.938883 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:47.981463 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.981436 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 20:46:47.981590 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:47.981502 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gnp52" Nov 03 20:46:48.000544 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.000504 2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal"] Nov 03 20:46:48.000661 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.000592 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:46:48.001552 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.001531 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:46:48.001625 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.001564 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:46:48.001625 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.001574 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:46:48.003572 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.003557 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:46:48.003748 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.003731 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:48.003790 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.003767 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:46:48.004406 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.004387 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:46:48.004486 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.004419 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:46:48.004486 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.004429 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:46:48.004486 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.004397 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:46:48.004486 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.004481 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:46:48.004605 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.004496 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:46:48.006419 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.006403 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:48.006463 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.006442 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:46:48.007208 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.007194 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:46:48.007285 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.007219 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:46:48.007285 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.007230 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:46:48.029060 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:48.029041 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-134.us-west-2.compute.internal\" not found" node="ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:48.032000 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:48.031982 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-134.us-west-2.compute.internal\" not found" node="ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:48.046031 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.046008 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:46:48.047656 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.047642 2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:46:48.064388 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.064365 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c672b05e20cfe6494452bceff1f2bac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal\" (UID: \"6c672b05e20cfe6494452bceff1f2bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:48.064463 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.064393 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c672b05e20cfe6494452bceff1f2bac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal\" (UID: \"6c672b05e20cfe6494452bceff1f2bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:48.064463 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.064412 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/954da78189a5577b346472d086419753-config\") pod \"kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal\" (UID: \"954da78189a5577b346472d086419753\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:48.139018 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.138981 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:46:48.142088 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.141998 2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:48.142088 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:48.142051 2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-134.us-west-2.compute.internal\": node \"ip-10-0-130-134.us-west-2.compute.internal\" not found" Nov 03 20:46:48.142280 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.142189 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:46:48.162058 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.162020 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:48.164874 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.164853 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c672b05e20cfe6494452bceff1f2bac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal\" (UID: \"6c672b05e20cfe6494452bceff1f2bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:48.164960 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.164883 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c672b05e20cfe6494452bceff1f2bac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal\" (UID: \"6c672b05e20cfe6494452bceff1f2bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:48.164960 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.164921 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/954da78189a5577b346472d086419753-config\") pod \"kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal\" (UID: \"954da78189a5577b346472d086419753\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:48.165026 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.164972 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/954da78189a5577b346472d086419753-config\") pod \"kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal\" (UID: \"954da78189a5577b346472d086419753\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:48.165026 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.164976 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c672b05e20cfe6494452bceff1f2bac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal\" (UID: \"6c672b05e20cfe6494452bceff1f2bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:48.165026 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.164972 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c672b05e20cfe6494452bceff1f2bac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal\" (UID: \"6c672b05e20cfe6494452bceff1f2bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:48.268698 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.268616 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:46:48.268840 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.268743 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:48.324633 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.324605 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:46:48.332760 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.332729 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:48.334920 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.334878 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:46:48.695767 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.695515 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 20:46:48.696376 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.695848 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 20:46:48.696376 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.695848 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 20:46:48.696376 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:48.695944 2583 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a7b0dd8fedf904086887a3a80115b944-867db6f534d550a3.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-130-134.us-west-2.compute.internal.18e4a3f129ede347\": read tcp 10.0.130.134:45510->18.220.211.213:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-130-134.us-west-2.compute.internal.18e4a3f129ede347 default 4821 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-134.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-134.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-134.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:46:47 +0000 UTC,LastTimestamp:2026-11-03 20:46:47.938860108 +0000 UTC m=+0.539853147,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-134.us-west-2.compute.internal,}" Nov 03 20:46:48.736761 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.736712 2583 apiserver.go:52] "Watching apiserver" Nov 03 20:46:48.762355 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.762321 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 20:46:48.985785 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.985672 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:41:48 +0000 UTC" deadline="2028-06-27 17:15:58.721205244 +0000 UTC" Nov 03 20:46:48.985785 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:48.985712 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14444h29m9.735496076s" Nov 03 20:46:49.039758 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.039718 2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:46:49.040119 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.040084 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn","openshift-network-operator/iptables-alerter-wl57x","openshift-ovn-kubernetes/ovnkube-node-vkh7b","openshift-image-registry/node-ca-r9lld","kube-system/konnectivity-agent-j5mgz","openshift-cluster-node-tuning-operator/tuned-7m7vq","openshift-dns/node-resolver-sm2mn","openshift-multus/multus-snszh","kube-system/global-pull-secret-syncer-lvbhk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-tgl9l","openshift-multus/network-metrics-daemon-mr55c","openshift-network-diagnostics/network-check-target-l7lxn","kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal"] Nov 03 20:46:49.042573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.042551 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j5mgz" Nov 03 20:46:49.044645 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.044627 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-snszh" Nov 03 20:46:49.046866 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.046847 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sm2mn" Nov 03 20:46:49.049080 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.049063 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r9lld" Nov 03 20:46:49.051301 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.051283 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.053448 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.053433 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.055699 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.055683 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.058076 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.058057 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wl57x" Nov 03 20:46:49.061362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.061330 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:46:49.061750 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.061725 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lvbhk" podUID="f1eb3e05-8b22-4992-97d9-c71ecc6b99dc" Nov 03 20:46:49.064090 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.064073 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.066346 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.066322 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:46:49.066431 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.066386 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr55c" podUID="84556453-e5d9-4449-ba59-819c248e1e2a" Nov 03 20:46:49.068637 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.068620 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:46:49.068714 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.068671 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7lxn" podUID="af51f192-4170-4d3b-85ae-7d2ceb14de05" Nov 03 20:46:49.071216 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071194 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-device-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.071292 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071232 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cf6a6e00-44f7-4af0-be4d-3650c552d729-agent-certs\") pod \"konnectivity-agent-j5mgz\" (UID: \"cf6a6e00-44f7-4af0-be4d-3650c552d729\") " pod="kube-system/konnectivity-agent-j5mgz" Nov 03 20:46:49.071292 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071259 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-etc-kubernetes\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.071292 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071284 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-host\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.071409 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071346 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvp8m\" (UniqueName: \"kubernetes.io/projected/542dd56a-cfab-40e2-a01b-a172651d46f8-kube-api-access-cvp8m\") pod \"iptables-alerter-wl57x\" (UID: \"542dd56a-cfab-40e2-a01b-a172651d46f8\") " pod="openshift-network-operator/iptables-alerter-wl57x" Nov 03 20:46:49.071467 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071403 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-var-lib-cni-multus\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.071467 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071435 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxhlc\" (UniqueName: \"kubernetes.io/projected/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-kube-api-access-gxhlc\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.071571 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071470 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e456784-7491-426a-80e5-17a9dfa2f07f-ovnkube-script-lib\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.071571 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071492 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-sysconfig\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.071571 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071510 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-dbus\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:46:49.071571 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071529 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-systemd-units\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.071571 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071566 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-run-openvswitch\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.071825 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071599 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-sysctl-d\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.071825 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071628 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-run\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.071825 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071645 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c22-9550-4b1c-8134-fc80562025eb-tmp\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.071825 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071682 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-cni-binary-copy\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.071825 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071730 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.071825 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071763 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-cnibin\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.071825 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071789 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-var-lib-kubelet\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.071825 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071809 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-daemon-config\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.072282 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071836 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-kubelet\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.072282 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071853 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-run-ovn\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.072282 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071866 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-sys\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.072282 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071881 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:46:49.072282 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071936 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.072282 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071960 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-cni-binary-copy\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.072282 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.071979 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-lib-modules\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.072282 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072002 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-var-lib-kubelet\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.072282 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072025 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cf6a6e00-44f7-4af0-be4d-3650c552d729-konnectivity-ca\") pod \"konnectivity-agent-j5mgz\" (UID: \"cf6a6e00-44f7-4af0-be4d-3650c552d729\") " pod="kube-system/konnectivity-agent-j5mgz" Nov 03 20:46:49.072282 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072041 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-cni-dir\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.072282 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072057 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-run-k8s-cni-cncf-io\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.072282 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072071 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mzjv\" (UniqueName: \"kubernetes.io/projected/eafb0c22-9550-4b1c-8134-fc80562025eb-kube-api-access-2mzjv\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.072282 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072093 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-var-lib-openvswitch\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.072282 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072120 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-cni-netd\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.072282 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072145 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.072282 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072164 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e456784-7491-426a-80e5-17a9dfa2f07f-ovnkube-config\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.072814 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072200 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7559e5a-a8fd-450c-8f40-f2072e93a9b5-tmp-dir\") pod \"node-resolver-sm2mn\" (UID: \"c7559e5a-a8fd-450c-8f40-f2072e93a9b5\") " pod="openshift-dns/node-resolver-sm2mn" Nov 03 20:46:49.072814 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072219 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-system-cni-dir\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.072814 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072234 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e456784-7491-426a-80e5-17a9dfa2f07f-env-overrides\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.072814 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072252 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e456784-7491-426a-80e5-17a9dfa2f07f-ovn-node-metrics-cert\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.072814 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072269 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-modprobe-d\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.072814 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072302 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-sysctl-conf\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.072814 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072324 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-tuned\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.072814 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072342 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-registration-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.072814 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072361 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-sys-fs\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.072814 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072394 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/542dd56a-cfab-40e2-a01b-a172651d46f8-host-slash\") pod \"iptables-alerter-wl57x\" (UID: \"542dd56a-cfab-40e2-a01b-a172651d46f8\") " pod="openshift-network-operator/iptables-alerter-wl57x" Nov 03 20:46:49.072814 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072430 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-run-systemd\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.072814 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072462 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.072814 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072492 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.072814 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072518 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.072814 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072549 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbfzh\" (UniqueName: \"kubernetes.io/projected/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-kube-api-access-dbfzh\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.072814 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072575 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-cni-bin\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.073362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072600 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7n4p\" (UniqueName: \"kubernetes.io/projected/8e456784-7491-426a-80e5-17a9dfa2f07f-kube-api-access-v7n4p\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.073362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072626 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-kubernetes\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.073362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072661 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-systemd\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.073362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072687 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-var-lib-cni-bin\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.073362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072740 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-conf-dir\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.073362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072765 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc6rc\" (UniqueName: \"kubernetes.io/projected/462e871a-1226-4f07-b34c-ef612fa6bf52-kube-api-access-pc6rc\") pod \"node-ca-r9lld\" (UID: \"462e871a-1226-4f07-b34c-ef612fa6bf52\") " pod="openshift-image-registry/node-ca-r9lld" Nov 03 20:46:49.073362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072783 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-slash\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.073362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072813 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t42hf\" (UniqueName: \"kubernetes.io/projected/c7559e5a-a8fd-450c-8f40-f2072e93a9b5-kube-api-access-t42hf\") pod \"node-resolver-sm2mn\" (UID: \"c7559e5a-a8fd-450c-8f40-f2072e93a9b5\") " pod="openshift-dns/node-resolver-sm2mn" Nov 03 20:46:49.073362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072829 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-hostroot\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.073362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072845 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/462e871a-1226-4f07-b34c-ef612fa6bf52-host\") pod \"node-ca-r9lld\" (UID: \"462e871a-1226-4f07-b34c-ef612fa6bf52\") " pod="openshift-image-registry/node-ca-r9lld" Nov 03 20:46:49.073362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072860 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/462e871a-1226-4f07-b34c-ef612fa6bf52-serviceca\") pod \"node-ca-r9lld\" (UID: \"462e871a-1226-4f07-b34c-ef612fa6bf52\") " pod="openshift-image-registry/node-ca-r9lld" Nov 03 20:46:49.073362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072883 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-log-socket\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.073362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072950 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/542dd56a-cfab-40e2-a01b-a172651d46f8-iptables-alerter-script\") pod \"iptables-alerter-wl57x\" (UID: \"542dd56a-cfab-40e2-a01b-a172651d46f8\") " pod="openshift-network-operator/iptables-alerter-wl57x" Nov 03 20:46:49.073362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.072988 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7559e5a-a8fd-450c-8f40-f2072e93a9b5-hosts-file\") pod \"node-resolver-sm2mn\" (UID: \"c7559e5a-a8fd-450c-8f40-f2072e93a9b5\") " pod="openshift-dns/node-resolver-sm2mn" Nov 03 20:46:49.073362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.073032 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-socket-dir-parent\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.073362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.073051 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-run-netns\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.073362 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.073069 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-socket-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.073910 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.073086 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76nsp\" (UniqueName: \"kubernetes.io/projected/245acdd3-cebd-49fe-bf1e-867df120ac8e-kube-api-access-76nsp\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.073910 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.073111 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-system-cni-dir\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.073910 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.073126 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.073910 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.073149 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-run-netns\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.073910 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.073178 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-etc-openvswitch\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.073910 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.073194 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-node-log\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.073910 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.073214 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-kubelet-config\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:46:49.073910 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.073235 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-cnibin\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.073910 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.073286 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-run-multus-certs\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.100749 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.100721 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:46:49.100906 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.100860 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gf6gt\"" Nov 03 20:46:49.100989 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.100920 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:46:49.102243 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.102217 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ldspv\"" Nov 03 20:46:49.102344 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.102271 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:46:49.102344 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.102310 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:46:49.102453 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.102389 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:46:49.102453 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.102420 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:46:49.104290 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.104272 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:46:49.104502 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.104484 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2g8j2\"" Nov 03 20:46:49.104568 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.104516 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:46:49.107103 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.107088 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:46:49.107298 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.107284 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:46:49.107342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.107288 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:46:49.107815 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.107800 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-868xx\"" Nov 03 20:46:49.109253 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.109238 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:46:49.109345 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.109331 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xldt8\"" Nov 03 20:46:49.109384 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.109353 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:46:49.121140 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.121120 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:46:49.153460 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.153433 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:46:49.153460 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.153434 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:46:49.153675 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.153435 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:46:49.153675 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.153475 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 20:46:49.153675 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.153603 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:46:49.155933 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.155914 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7vv6z\"" Nov 03 20:46:49.156131 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.156116 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:46:49.156205 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.156116 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vwxfp\"" Nov 03 20:46:49.156205 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.156180 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:46:49.156505 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.156204 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:46:49.156505 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.156274 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:46:49.156505 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.156313 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tlf9q\"" Nov 03 20:46:49.156505 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.156388 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:46:49.156505 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.156398 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:46:49.156505 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.156480 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 20:46:49.156805 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.156518 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4w7dg" Nov 03 20:46:49.156805 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.156570 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r6str\"" Nov 03 20:46:49.156805 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.156749 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:46:49.157177 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.157160 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:46:49.164096 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.163656 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 20:46:49.173689 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173659 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.173818 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173698 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.173818 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173717 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.173818 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173736 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbfzh\" (UniqueName: \"kubernetes.io/projected/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-kube-api-access-dbfzh\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.173818 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173759 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-cni-bin\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.173818 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173763 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.173818 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173783 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7n4p\" (UniqueName: \"kubernetes.io/projected/8e456784-7491-426a-80e5-17a9dfa2f07f-kube-api-access-v7n4p\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.173818 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173790 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.173818 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173804 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-kubernetes\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.174330 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173826 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-systemd\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.174330 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173850 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-var-lib-cni-bin\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.174330 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173875 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-conf-dir\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.174330 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173877 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.174330 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173927 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-conf-dir\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.174330 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173945 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-systemd\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.174330 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173954 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-var-lib-cni-bin\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.174330 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173972 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6rc\" (UniqueName: \"kubernetes.io/projected/462e871a-1226-4f07-b34c-ef612fa6bf52-kube-api-access-pc6rc\") pod \"node-ca-r9lld\" (UID: \"462e871a-1226-4f07-b34c-ef612fa6bf52\") " pod="openshift-image-registry/node-ca-r9lld" Nov 03 20:46:49.174330 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173976 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-kubernetes\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.174330 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.173993 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-slash\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.174330 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174024 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-slash\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.174330 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174077 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-cni-bin\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.174330 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174115 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t42hf\" (UniqueName: \"kubernetes.io/projected/c7559e5a-a8fd-450c-8f40-f2072e93a9b5-kube-api-access-t42hf\") pod \"node-resolver-sm2mn\" (UID: \"c7559e5a-a8fd-450c-8f40-f2072e93a9b5\") " pod="openshift-dns/node-resolver-sm2mn" Nov 03 20:46:49.174330 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174142 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-hostroot\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.174330 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174164 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/462e871a-1226-4f07-b34c-ef612fa6bf52-host\") pod \"node-ca-r9lld\" (UID: \"462e871a-1226-4f07-b34c-ef612fa6bf52\") " pod="openshift-image-registry/node-ca-r9lld" Nov 03 20:46:49.174330 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174186 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/462e871a-1226-4f07-b34c-ef612fa6bf52-serviceca\") pod \"node-ca-r9lld\" (UID: \"462e871a-1226-4f07-b34c-ef612fa6bf52\") " pod="openshift-image-registry/node-ca-r9lld" Nov 03 20:46:49.174330 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174211 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-log-socket\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.174330 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174217 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/462e871a-1226-4f07-b34c-ef612fa6bf52-host\") pod \"node-ca-r9lld\" (UID: \"462e871a-1226-4f07-b34c-ef612fa6bf52\") " pod="openshift-image-registry/node-ca-r9lld" Nov 03 20:46:49.175141 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174219 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-hostroot\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.175141 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174251 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/542dd56a-cfab-40e2-a01b-a172651d46f8-iptables-alerter-script\") pod \"iptables-alerter-wl57x\" (UID: \"542dd56a-cfab-40e2-a01b-a172651d46f8\") " pod="openshift-network-operator/iptables-alerter-wl57x" Nov 03 20:46:49.175141 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174278 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7559e5a-a8fd-450c-8f40-f2072e93a9b5-hosts-file\") pod \"node-resolver-sm2mn\" (UID: \"c7559e5a-a8fd-450c-8f40-f2072e93a9b5\") " pod="openshift-dns/node-resolver-sm2mn" Nov 03 20:46:49.175141 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174285 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-log-socket\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.175141 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174305 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-socket-dir-parent\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.175141 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174332 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-run-netns\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.175141 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174340 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7559e5a-a8fd-450c-8f40-f2072e93a9b5-hosts-file\") pod \"node-resolver-sm2mn\" (UID: \"c7559e5a-a8fd-450c-8f40-f2072e93a9b5\") " pod="openshift-dns/node-resolver-sm2mn" Nov 03 20:46:49.175141 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174359 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-socket-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.175141 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174400 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-socket-dir-parent\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.175141 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174404 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-run-netns\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.175141 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174446 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76nsp\" (UniqueName: \"kubernetes.io/projected/245acdd3-cebd-49fe-bf1e-867df120ac8e-kube-api-access-76nsp\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.175141 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174471 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-system-cni-dir\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.175141 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174497 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.175141 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174518 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-system-cni-dir\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.175141 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174522 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-run-netns\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.175141 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174549 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-etc-openvswitch\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.175141 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174573 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-node-log\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.175844 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174601 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:46:49.175844 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174630 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-kubelet-config\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:46:49.175844 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174655 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-cnibin\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.175844 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174681 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-run-multus-certs\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.175844 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174705 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-device-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.175844 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174735 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cf6a6e00-44f7-4af0-be4d-3650c552d729-agent-certs\") pod \"konnectivity-agent-j5mgz\" (UID: \"cf6a6e00-44f7-4af0-be4d-3650c552d729\") " pod="kube-system/konnectivity-agent-j5mgz" Nov 03 20:46:49.175844 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174762 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-etc-kubernetes\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.175844 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174785 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-host\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.175844 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174813 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvp8m\" (UniqueName: \"kubernetes.io/projected/542dd56a-cfab-40e2-a01b-a172651d46f8-kube-api-access-cvp8m\") pod \"iptables-alerter-wl57x\" (UID: \"542dd56a-cfab-40e2-a01b-a172651d46f8\") " pod="openshift-network-operator/iptables-alerter-wl57x" Nov 03 20:46:49.175844 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174837 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-var-lib-cni-multus\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.175844 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174860 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxhlc\" (UniqueName: \"kubernetes.io/projected/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-kube-api-access-gxhlc\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.175844 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174886 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e456784-7491-426a-80e5-17a9dfa2f07f-ovnkube-script-lib\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.175844 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174911 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-sysconfig\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.175844 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174935 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-dbus\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:46:49.175844 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.174979 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-systemd-units\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.175844 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175006 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-run-openvswitch\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.175844 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175015 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-device-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.176573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175031 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-sysctl-d\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.176573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175055 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-run\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.176573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175059 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-run-netns\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.176573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175080 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c22-9550-4b1c-8134-fc80562025eb-tmp\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.176573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175087 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-etc-openvswitch\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.176573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175108 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vqts\" (UniqueName: \"kubernetes.io/projected/84556453-e5d9-4449-ba59-819c248e1e2a-kube-api-access-4vqts\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:46:49.176573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175114 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-node-log\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.176573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175149 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-cni-binary-copy\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.176573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175169 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.176573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175188 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-cnibin\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.176573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175203 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-var-lib-kubelet\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.176573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175219 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-daemon-config\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.176573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175236 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-kubelet\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.176573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175255 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-run-ovn\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.176573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175270 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-sys\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.176573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175286 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:46:49.176573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175303 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.177275 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175320 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-cni-binary-copy\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.177275 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175338 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-lib-modules\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.177275 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175353 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-var-lib-kubelet\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.177275 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175353 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/462e871a-1226-4f07-b34c-ef612fa6bf52-serviceca\") pod \"node-ca-r9lld\" (UID: \"462e871a-1226-4f07-b34c-ef612fa6bf52\") " pod="openshift-image-registry/node-ca-r9lld" Nov 03 20:46:49.177275 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175371 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cf6a6e00-44f7-4af0-be4d-3650c552d729-konnectivity-ca\") pod \"konnectivity-agent-j5mgz\" (UID: \"cf6a6e00-44f7-4af0-be4d-3650c552d729\") " pod="kube-system/konnectivity-agent-j5mgz" Nov 03 20:46:49.177275 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175390 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-cni-dir\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.177275 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175406 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-run-k8s-cni-cncf-io\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.177275 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175423 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mzjv\" (UniqueName: \"kubernetes.io/projected/eafb0c22-9550-4b1c-8134-fc80562025eb-kube-api-access-2mzjv\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.177275 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175455 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd729\" (UniqueName: \"kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729\") pod \"network-check-target-l7lxn\" (UID: \"af51f192-4170-4d3b-85ae-7d2ceb14de05\") " pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:46:49.177275 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175482 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-var-lib-openvswitch\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.177275 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175486 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-sysconfig\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.177275 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175526 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-cni-netd\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.177275 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175530 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-var-lib-openvswitch\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.177275 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175544 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-socket-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.177275 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175547 2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 20:46:49.177275 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175559 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.177275 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175600 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-run-multus-certs\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.178156 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175604 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/542dd56a-cfab-40e2-a01b-a172651d46f8-iptables-alerter-script\") pod \"iptables-alerter-wl57x\" (UID: \"542dd56a-cfab-40e2-a01b-a172651d46f8\") " pod="openshift-network-operator/iptables-alerter-wl57x" Nov 03 20:46:49.178156 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175611 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e456784-7491-426a-80e5-17a9dfa2f07f-ovnkube-config\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.178156 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175613 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.178156 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175603 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-kubelet-config\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:46:49.178156 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175638 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-cnibin\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.178156 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175658 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-cnibin\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.178156 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175701 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-var-lib-kubelet\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.178156 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175701 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-etc-kubernetes\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.178156 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175728 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-sysctl-d\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.178156 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175746 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-host\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.178156 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175785 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-cni-netd\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.178156 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175824 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.178156 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.175941 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-lib-modules\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.178156 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176008 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-kubelet\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.178156 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176052 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-var-lib-cni-multus\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.178156 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176122 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-sys\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.178156 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176055 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-run-ovn\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.178156 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.176142 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:46:49.179077 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176209 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-systemd-units\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.179077 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.176236 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret podName:f1eb3e05-8b22-4992-97d9-c71ecc6b99dc nodeName:}" failed. No retries permitted until 2026-11-03 20:46:49.67619349 +0000 UTC m=+2.277186516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret") pod "global-pull-secret-syncer-lvbhk" (UID: "f1eb3e05-8b22-4992-97d9-c71ecc6b99dc") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:46:49.179077 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176266 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-dbus\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:46:49.179077 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176313 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-run\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.179077 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176330 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-daemon-config\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.179077 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176330 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-cni-binary-copy\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.179077 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176385 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-run-openvswitch\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.179077 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176412 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-var-lib-kubelet\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.179077 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176417 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7559e5a-a8fd-450c-8f40-f2072e93a9b5-tmp-dir\") pod \"node-resolver-sm2mn\" (UID: \"c7559e5a-a8fd-450c-8f40-f2072e93a9b5\") " pod="openshift-dns/node-resolver-sm2mn" Nov 03 20:46:49.179077 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176449 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-system-cni-dir\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.179077 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176475 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e456784-7491-426a-80e5-17a9dfa2f07f-env-overrides\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.179077 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176501 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e456784-7491-426a-80e5-17a9dfa2f07f-ovn-node-metrics-cert\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.179077 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176525 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-modprobe-d\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.179077 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176549 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-sysctl-conf\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.179077 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176552 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e456784-7491-426a-80e5-17a9dfa2f07f-ovnkube-config\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.179077 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176574 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-tuned\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.179077 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176600 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-registration-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.179913 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176625 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-sys-fs\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.179913 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176650 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/542dd56a-cfab-40e2-a01b-a172651d46f8-host-slash\") pod \"iptables-alerter-wl57x\" (UID: \"542dd56a-cfab-40e2-a01b-a172651d46f8\") " pod="openshift-network-operator/iptables-alerter-wl57x" Nov 03 20:46:49.179913 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176672 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-run-systemd\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.179913 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176753 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-run-systemd\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.179913 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176788 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.179913 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176801 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-cni-binary-copy\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.179913 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.176998 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e456784-7491-426a-80e5-17a9dfa2f07f-env-overrides\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.179913 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.177064 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-system-cni-dir\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.179913 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.177091 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cf6a6e00-44f7-4af0-be4d-3650c552d729-konnectivity-ca\") pod \"konnectivity-agent-j5mgz\" (UID: \"cf6a6e00-44f7-4af0-be4d-3650c552d729\") " pod="kube-system/konnectivity-agent-j5mgz" Nov 03 20:46:49.179913 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.177117 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-run-k8s-cni-cncf-io\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.179913 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.177152 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7559e5a-a8fd-450c-8f40-f2072e93a9b5-tmp-dir\") pod \"node-resolver-sm2mn\" (UID: \"c7559e5a-a8fd-450c-8f40-f2072e93a9b5\") " pod="openshift-dns/node-resolver-sm2mn" Nov 03 20:46:49.179913 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.177242 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-modprobe-d\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.179913 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.177274 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.179913 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.177293 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-sysctl-conf\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.179913 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.177307 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-sys-fs\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.179913 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.177341 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/542dd56a-cfab-40e2-a01b-a172651d46f8-host-slash\") pod \"iptables-alerter-wl57x\" (UID: \"542dd56a-cfab-40e2-a01b-a172651d46f8\") " pod="openshift-network-operator/iptables-alerter-wl57x" Nov 03 20:46:49.179913 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.177358 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-registration-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.180783 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.177455 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-cni-dir\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.180783 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.178198 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e456784-7491-426a-80e5-17a9dfa2f07f-ovnkube-script-lib\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.180783 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.179713 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e456784-7491-426a-80e5-17a9dfa2f07f-ovn-node-metrics-cert\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.180783 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.179729 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c22-9550-4b1c-8134-fc80562025eb-tmp\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.180783 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.180458 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-tuned\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.180783 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.180589 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cf6a6e00-44f7-4af0-be4d-3650c552d729-agent-certs\") pod \"konnectivity-agent-j5mgz\" (UID: \"cf6a6e00-44f7-4af0-be4d-3650c552d729\") " pod="kube-system/konnectivity-agent-j5mgz" Nov 03 20:46:49.241881 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.241813 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7n4p\" (UniqueName: \"kubernetes.io/projected/8e456784-7491-426a-80e5-17a9dfa2f07f-kube-api-access-v7n4p\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.242060 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.242033 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc6rc\" (UniqueName: \"kubernetes.io/projected/462e871a-1226-4f07-b34c-ef612fa6bf52-kube-api-access-pc6rc\") pod \"node-ca-r9lld\" (UID: \"462e871a-1226-4f07-b34c-ef612fa6bf52\") " pod="openshift-image-registry/node-ca-r9lld" Nov 03 20:46:49.242634 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.242579 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxhlc\" (UniqueName: \"kubernetes.io/projected/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-kube-api-access-gxhlc\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:46:49.242634 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.242589 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvp8m\" (UniqueName: \"kubernetes.io/projected/542dd56a-cfab-40e2-a01b-a172651d46f8-kube-api-access-cvp8m\") pod \"iptables-alerter-wl57x\" (UID: \"542dd56a-cfab-40e2-a01b-a172651d46f8\") " pod="openshift-network-operator/iptables-alerter-wl57x" Nov 03 20:46:49.242634 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.242602 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbfzh\" (UniqueName: \"kubernetes.io/projected/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-kube-api-access-dbfzh\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.242634 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.242577 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t42hf\" (UniqueName: \"kubernetes.io/projected/c7559e5a-a8fd-450c-8f40-f2072e93a9b5-kube-api-access-t42hf\") pod \"node-resolver-sm2mn\" (UID: \"c7559e5a-a8fd-450c-8f40-f2072e93a9b5\") " pod="openshift-dns/node-resolver-sm2mn" Nov 03 20:46:49.242857 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.242609 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76nsp\" (UniqueName: \"kubernetes.io/projected/245acdd3-cebd-49fe-bf1e-867df120ac8e-kube-api-access-76nsp\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.244233 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.244215 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mzjv\" (UniqueName: \"kubernetes.io/projected/eafb0c22-9550-4b1c-8134-fc80562025eb-kube-api-access-2mzjv\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.277231 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.277189 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:46:49.277362 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.277291 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:46:49.277362 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.277353 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs podName:84556453-e5d9-4449-ba59-819c248e1e2a nodeName:}" failed. No retries permitted until 2026-11-03 20:46:49.777332719 +0000 UTC m=+2.378325760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs") pod "network-metrics-daemon-mr55c" (UID: "84556453-e5d9-4449-ba59-819c248e1e2a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:46:49.277520 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.277409 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vqts\" (UniqueName: \"kubernetes.io/projected/84556453-e5d9-4449-ba59-819c248e1e2a-kube-api-access-4vqts\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:46:49.277577 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.277558 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd729\" (UniqueName: \"kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729\") pod \"network-check-target-l7lxn\" (UID: \"af51f192-4170-4d3b-85ae-7d2ceb14de05\") " pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:46:49.342790 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.342756 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:46:49.342790 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.342784 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:46:49.342790 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.342795 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rd729 for pod openshift-network-diagnostics/network-check-target-l7lxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:46:49.343035 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.342856 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729 podName:af51f192-4170-4d3b-85ae-7d2ceb14de05 nodeName:}" failed. No retries permitted until 2026-11-03 20:46:49.842840326 +0000 UTC m=+2.443833364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rd729" (UniqueName: "kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729") pod "network-check-target-l7lxn" (UID: "af51f192-4170-4d3b-85ae-7d2ceb14de05") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:46:49.345205 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.345173 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vqts\" (UniqueName: \"kubernetes.io/projected/84556453-e5d9-4449-ba59-819c248e1e2a-kube-api-access-4vqts\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:46:49.372631 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.372606 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j5mgz" Nov 03 20:46:49.383342 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.383317 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:46:49.405940 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.405897 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-snszh" Nov 03 20:46:49.421139 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.421106 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sm2mn" Nov 03 20:46:49.435256 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.435232 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r9lld" Nov 03 20:46:49.468697 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.468672 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:46:49.478403 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.478379 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:46:49.485525 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.485502 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wl57x" Nov 03 20:46:49.504666 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.504597 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:46:49.658104 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:49.658059 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c672b05e20cfe6494452bceff1f2bac.slice/crio-7069e383c5e90e1f48fc0dce4f850b630e915add3755f25704bc7093cbdeae28 WatchSource:0}: Error finding container 7069e383c5e90e1f48fc0dce4f850b630e915add3755f25704bc7093cbdeae28: Status 404 returned error can't find the container with id 7069e383c5e90e1f48fc0dce4f850b630e915add3755f25704bc7093cbdeae28 Nov 03 20:46:49.663191 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.663172 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:46:49.669614 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.669595 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:46:49.680113 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.680091 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:46:49.680217 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.680205 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:46:49.680264 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.680253 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret podName:f1eb3e05-8b22-4992-97d9-c71ecc6b99dc nodeName:}" failed. No retries permitted until 2026-11-03 20:46:50.680240821 +0000 UTC m=+3.281233846 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret") pod "global-pull-secret-syncer-lvbhk" (UID: "f1eb3e05-8b22-4992-97d9-c71ecc6b99dc") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:46:49.704380 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:49.704357 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954da78189a5577b346472d086419753.slice/crio-f2cf2855e1e18cbff8c7c4f4ff8c2b8b63020f9b606a416adfd2f86609337bb7 WatchSource:0}: Error finding container f2cf2855e1e18cbff8c7c4f4ff8c2b8b63020f9b606a416adfd2f86609337bb7: Status 404 returned error can't find the container with id f2cf2855e1e18cbff8c7c4f4ff8c2b8b63020f9b606a416adfd2f86609337bb7 Nov 03 20:46:49.781296 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.781201 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:46:49.781461 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.781334 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:46:49.781461 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.781401 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs podName:84556453-e5d9-4449-ba59-819c248e1e2a nodeName:}" failed. No retries permitted until 2026-11-03 20:46:50.781382351 +0000 UTC m=+3.382375390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs") pod "network-metrics-daemon-mr55c" (UID: "84556453-e5d9-4449-ba59-819c248e1e2a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:46:49.882385 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.882354 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd729\" (UniqueName: \"kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729\") pod \"network-check-target-l7lxn\" (UID: \"af51f192-4170-4d3b-85ae-7d2ceb14de05\") " pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:46:49.882587 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.882517 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:46:49.882587 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.882558 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:46:49.882587 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.882569 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rd729 for pod openshift-network-diagnostics/network-check-target-l7lxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:46:49.882727 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:49.882624 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729 podName:af51f192-4170-4d3b-85ae-7d2ceb14de05 nodeName:}" failed. No retries permitted until 2026-11-03 20:46:50.882606347 +0000 UTC m=+3.483599371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd729" (UniqueName: "kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729") pod "network-check-target-l7lxn" (UID: "af51f192-4170-4d3b-85ae-7d2ceb14de05") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:46:49.906421 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.906364 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal" event={"ID":"954da78189a5577b346472d086419753","Type":"ContainerStarted","Data":"f2cf2855e1e18cbff8c7c4f4ff8c2b8b63020f9b606a416adfd2f86609337bb7"} Nov 03 20:46:49.907361 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:49.907339 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" event={"ID":"6c672b05e20cfe6494452bceff1f2bac","Type":"ContainerStarted","Data":"7069e383c5e90e1f48fc0dce4f850b630e915add3755f25704bc7093cbdeae28"} Nov 03 20:46:50.160462 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:50.160341 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:41:49 +0000 UTC" deadline="2028-07-27 05:04:51.085552388 +0000 UTC" Nov 03 20:46:50.160462 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:50.160406 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15152h18m0.925150384s" Nov 03 20:46:50.164551 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:50.164530 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:46:50.324387 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:50.324353 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6a6e00_44f7_4af0_be4d_3650c552d729.slice/crio-0a3fe3cd0d75872773797b98a329cbc0c746f6c3f127fbfb1f4c1a7a3a052a95 WatchSource:0}: Error finding container 0a3fe3cd0d75872773797b98a329cbc0c746f6c3f127fbfb1f4c1a7a3a052a95: Status 404 returned error can't find the container with id 0a3fe3cd0d75872773797b98a329cbc0c746f6c3f127fbfb1f4c1a7a3a052a95 Nov 03 20:46:50.364508 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:50.364471 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245acdd3_cebd_49fe_bf1e_867df120ac8e.slice/crio-4a7aa36d62284423b604b802a6c71bd13958257e15869924b627cd54257f9f71 WatchSource:0}: Error finding container 4a7aa36d62284423b604b802a6c71bd13958257e15869924b627cd54257f9f71: Status 404 returned error can't find the container with id 4a7aa36d62284423b604b802a6c71bd13958257e15869924b627cd54257f9f71 Nov 03 20:46:50.398654 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:50.398624 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462e871a_1226_4f07_b34c_ef612fa6bf52.slice/crio-38655434f2520c5e2917c6fae8d88b6f7b7fa718d8f5deeedcf30cef0b89c243 WatchSource:0}: Error finding container 38655434f2520c5e2917c6fae8d88b6f7b7fa718d8f5deeedcf30cef0b89c243: Status 404 returned error can't find the container with id 38655434f2520c5e2917c6fae8d88b6f7b7fa718d8f5deeedcf30cef0b89c243 Nov 03 20:46:50.419354 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:50.419321 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7559e5a_a8fd_450c_8f40_f2072e93a9b5.slice/crio-a6991d2ce08040d5e55532dfe0c9cafcc01e539d91fed48c1a9d8073fdbcb46a WatchSource:0}: Error finding container a6991d2ce08040d5e55532dfe0c9cafcc01e539d91fed48c1a9d8073fdbcb46a: Status 404 returned error can't find the container with id a6991d2ce08040d5e55532dfe0c9cafcc01e539d91fed48c1a9d8073fdbcb46a Nov 03 20:46:50.455059 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:50.455026 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea105ed_0fa7_46a4_8d9f_94714d3a9b20.slice/crio-aff42e60564e77b412cc0588ce32d9ca1beef086c381cb9deefc67e35867fcea WatchSource:0}: Error finding container aff42e60564e77b412cc0588ce32d9ca1beef086c381cb9deefc67e35867fcea: Status 404 returned error can't find the container with id aff42e60564e77b412cc0588ce32d9ca1beef086c381cb9deefc67e35867fcea Nov 03 20:46:50.492093 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:50.492066 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e456784_7491_426a_80e5_17a9dfa2f07f.slice/crio-22fa6676ded0c91b28b32d11334dff7ae62d8127610991237d29edae9b8b5656 WatchSource:0}: Error finding container 22fa6676ded0c91b28b32d11334dff7ae62d8127610991237d29edae9b8b5656: Status 404 returned error can't find the container with id 22fa6676ded0c91b28b32d11334dff7ae62d8127610991237d29edae9b8b5656 Nov 03 20:46:50.546080 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:50.546047 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542dd56a_cfab_40e2_a01b_a172651d46f8.slice/crio-2227b91036d9447bab326f0fffe436b5f507fac3ea16c99420efffa4cc903790 WatchSource:0}: Error finding container 2227b91036d9447bab326f0fffe436b5f507fac3ea16c99420efffa4cc903790: Status 404 returned error can't find the container with id 2227b91036d9447bab326f0fffe436b5f507fac3ea16c99420efffa4cc903790 Nov 03 20:46:50.556040 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:50.556013 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb02f36e_58c6_47fb_b6f7_0ba78d029ffb.slice/crio-7ea1e10845bd2bf33c03c886f2bd6ed20c988bfc836f634567dd107aa474ac20 WatchSource:0}: Error finding container 7ea1e10845bd2bf33c03c886f2bd6ed20c988bfc836f634567dd107aa474ac20: Status 404 returned error can't find the container with id 7ea1e10845bd2bf33c03c886f2bd6ed20c988bfc836f634567dd107aa474ac20 Nov 03 20:46:50.687545 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:50.687451 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:46:50.687703 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:50.687583 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:46:50.687703 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:50.687638 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret podName:f1eb3e05-8b22-4992-97d9-c71ecc6b99dc nodeName:}" failed. No retries permitted until 2026-11-03 20:46:52.687625676 +0000 UTC m=+5.288618701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret") pod "global-pull-secret-syncer-lvbhk" (UID: "f1eb3e05-8b22-4992-97d9-c71ecc6b99dc") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:46:50.788813 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:50.788779 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:46:50.789332 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:50.788896 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:46:50.789332 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:50.788970 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs podName:84556453-e5d9-4449-ba59-819c248e1e2a nodeName:}" failed. No retries permitted until 2026-11-03 20:46:52.788952771 +0000 UTC m=+5.389945805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs") pod "network-metrics-daemon-mr55c" (UID: "84556453-e5d9-4449-ba59-819c248e1e2a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:46:50.806944 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:46:50.806915 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafb0c22_9550_4b1c_8134_fc80562025eb.slice/crio-0e832050e0f0c54e223aff7b4bb6633c8b0a339967703e1499541b2fb5a44947 WatchSource:0}: Error finding container 0e832050e0f0c54e223aff7b4bb6633c8b0a339967703e1499541b2fb5a44947: Status 404 returned error can't find the container with id 0e832050e0f0c54e223aff7b4bb6633c8b0a339967703e1499541b2fb5a44947 Nov 03 20:46:50.890181 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:50.890137 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd729\" (UniqueName: \"kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729\") pod \"network-check-target-l7lxn\" (UID: \"af51f192-4170-4d3b-85ae-7d2ceb14de05\") " pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:46:50.890352 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:50.890336 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:46:50.890400 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:50.890359 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:46:50.890400 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:50.890370 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rd729 for pod openshift-network-diagnostics/network-check-target-l7lxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:46:50.890463 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:50.890427 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729 podName:af51f192-4170-4d3b-85ae-7d2ceb14de05 nodeName:}" failed. No retries permitted until 2026-11-03 20:46:52.890413087 +0000 UTC m=+5.491406112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd729" (UniqueName: "kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729") pod "network-check-target-l7lxn" (UID: "af51f192-4170-4d3b-85ae-7d2ceb14de05") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:46:50.900347 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:50.900312 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:46:50.900501 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:50.900314 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:46:50.900501 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:50.900422 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7lxn" podUID="af51f192-4170-4d3b-85ae-7d2ceb14de05" Nov 03 20:46:50.900501 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:50.900321 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:46:50.900630 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:50.900498 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lvbhk" podUID="f1eb3e05-8b22-4992-97d9-c71ecc6b99dc" Nov 03 20:46:50.900630 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:50.900589 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr55c" podUID="84556453-e5d9-4449-ba59-819c248e1e2a" Nov 03 20:46:50.910791 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:50.910755 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wl57x" event={"ID":"542dd56a-cfab-40e2-a01b-a172651d46f8","Type":"ContainerStarted","Data":"2227b91036d9447bab326f0fffe436b5f507fac3ea16c99420efffa4cc903790"} Nov 03 20:46:50.911996 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:50.911968 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r9lld" event={"ID":"462e871a-1226-4f07-b34c-ef612fa6bf52","Type":"ContainerStarted","Data":"38655434f2520c5e2917c6fae8d88b6f7b7fa718d8f5deeedcf30cef0b89c243"} Nov 03 20:46:50.913254 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:50.913234 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" event={"ID":"245acdd3-cebd-49fe-bf1e-867df120ac8e","Type":"ContainerStarted","Data":"4a7aa36d62284423b604b802a6c71bd13958257e15869924b627cd54257f9f71"} Nov 03 20:46:50.914314 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:50.914286 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j5mgz" event={"ID":"cf6a6e00-44f7-4af0-be4d-3650c552d729","Type":"ContainerStarted","Data":"0a3fe3cd0d75872773797b98a329cbc0c746f6c3f127fbfb1f4c1a7a3a052a95"} Nov 03 20:46:50.915395 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:50.915375 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" event={"ID":"eafb0c22-9550-4b1c-8134-fc80562025eb","Type":"ContainerStarted","Data":"0e832050e0f0c54e223aff7b4bb6633c8b0a339967703e1499541b2fb5a44947"} Nov 03 20:46:50.916616 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:50.916595 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" event={"ID":"8e456784-7491-426a-80e5-17a9dfa2f07f","Type":"ContainerStarted","Data":"22fa6676ded0c91b28b32d11334dff7ae62d8127610991237d29edae9b8b5656"} Nov 03 20:46:50.917944 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:50.917907 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-snszh" event={"ID":"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20","Type":"ContainerStarted","Data":"aff42e60564e77b412cc0588ce32d9ca1beef086c381cb9deefc67e35867fcea"} Nov 03 20:46:50.919119 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:50.919097 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sm2mn" event={"ID":"c7559e5a-a8fd-450c-8f40-f2072e93a9b5","Type":"ContainerStarted","Data":"a6991d2ce08040d5e55532dfe0c9cafcc01e539d91fed48c1a9d8073fdbcb46a"} Nov 03 20:46:50.920150 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:50.920131 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tgl9l" event={"ID":"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb","Type":"ContainerStarted","Data":"7ea1e10845bd2bf33c03c886f2bd6ed20c988bfc836f634567dd107aa474ac20"} Nov 03 20:46:51.161067 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:51.160984 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:41:49 +0000 UTC" deadline="2028-05-03 19:24:27.134051134 +0000 UTC" Nov 03 20:46:51.161067 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:51.161020 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13126h37m35.973034951s" Nov 03 20:46:52.708334 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:52.708290 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:46:52.709002 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:52.708471 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:46:52.709002 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:52.708541 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret podName:f1eb3e05-8b22-4992-97d9-c71ecc6b99dc nodeName:}" failed. No retries permitted until 2026-11-03 20:46:56.70852015 +0000 UTC m=+9.309513177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret") pod "global-pull-secret-syncer-lvbhk" (UID: "f1eb3e05-8b22-4992-97d9-c71ecc6b99dc") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:46:52.809217 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:52.809122 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:46:52.809405 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:52.809334 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:46:52.809405 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:52.809404 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs podName:84556453-e5d9-4449-ba59-819c248e1e2a nodeName:}" failed. No retries permitted until 2026-11-03 20:46:56.809382156 +0000 UTC m=+9.410375182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs") pod "network-metrics-daemon-mr55c" (UID: "84556453-e5d9-4449-ba59-819c248e1e2a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:46:52.900658 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:52.900624 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:46:52.900840 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:52.900760 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7lxn" podUID="af51f192-4170-4d3b-85ae-7d2ceb14de05" Nov 03 20:46:52.900939 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:52.900844 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:46:52.901023 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:52.900926 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lvbhk" podUID="f1eb3e05-8b22-4992-97d9-c71ecc6b99dc" Nov 03 20:46:52.901023 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:52.901017 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:46:52.901170 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:52.901104 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr55c" podUID="84556453-e5d9-4449-ba59-819c248e1e2a" Nov 03 20:46:52.909594 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:52.909562 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd729\" (UniqueName: \"kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729\") pod \"network-check-target-l7lxn\" (UID: \"af51f192-4170-4d3b-85ae-7d2ceb14de05\") " pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:46:52.909777 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:52.909743 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:46:52.909777 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:52.909763 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:46:52.909777 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:52.909775 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rd729 for pod openshift-network-diagnostics/network-check-target-l7lxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:46:52.909966 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:52.909830 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729 podName:af51f192-4170-4d3b-85ae-7d2ceb14de05 nodeName:}" failed. No retries permitted until 2026-11-03 20:46:56.90981196 +0000 UTC m=+9.510804990 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd729" (UniqueName: "kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729") pod "network-check-target-l7lxn" (UID: "af51f192-4170-4d3b-85ae-7d2ceb14de05") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:46:54.900685 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:54.900410 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:46:54.901363 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:54.900827 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr55c" podUID="84556453-e5d9-4449-ba59-819c248e1e2a" Nov 03 20:46:54.901363 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:54.900900 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:46:54.901363 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:54.900989 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7lxn" podUID="af51f192-4170-4d3b-85ae-7d2ceb14de05" Nov 03 20:46:54.901363 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:54.901045 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:46:54.901363 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:54.901122 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lvbhk" podUID="f1eb3e05-8b22-4992-97d9-c71ecc6b99dc" Nov 03 20:46:54.953215 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:54.953174 2583 generic.go:356] "Generic (PLEG): container finished" podID="6c672b05e20cfe6494452bceff1f2bac" containerID="aafc8793f8121d4fb45fd3599a567b38ee778780783a6de3055486890eab708b" exitCode=0 Nov 03 20:46:54.953364 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:54.953276 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" event={"ID":"6c672b05e20cfe6494452bceff1f2bac","Type":"ContainerDied","Data":"aafc8793f8121d4fb45fd3599a567b38ee778780783a6de3055486890eab708b"} Nov 03 20:46:54.961838 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:54.961794 2583 generic.go:356] "Generic (PLEG): container finished" podID="fb02f36e-58c6-47fb-b6f7-0ba78d029ffb" containerID="6c525f10a9c77c8cb2fdd2be3dc983e4d260820427f09323cd79622ce92d1af0" exitCode=0 Nov 03 20:46:54.962164 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:54.962134 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tgl9l" event={"ID":"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb","Type":"ContainerDied","Data":"6c525f10a9c77c8cb2fdd2be3dc983e4d260820427f09323cd79622ce92d1af0"} Nov 03 20:46:54.968227 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:54.968195 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal" event={"ID":"954da78189a5577b346472d086419753","Type":"ContainerStarted","Data":"4d64fa7e8a094152280991f1982732bd4b514c71894808c8488261dbfbc60947"} Nov 03 20:46:55.218387 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:55.217484 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal" podStartSLOduration=6.217461936 podStartE2EDuration="6.217461936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:55.215361017 +0000 UTC m=+7.816354066" watchObservedRunningTime="2026-11-03 20:46:55.217461936 +0000 UTC m=+7.818454985" Nov 03 20:46:55.978011 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:55.977915 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j5mgz" event={"ID":"cf6a6e00-44f7-4af0-be4d-3650c552d729","Type":"ContainerStarted","Data":"dcd07a3bcfdab38f83350d0eb09a078c9a42eb0d4b2a326e3f3555156e0ce66b"} Nov 03 20:46:55.986023 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:55.985874 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" event={"ID":"6c672b05e20cfe6494452bceff1f2bac","Type":"ContainerStarted","Data":"99787d0dd813df80f11e4ae56e40c10cf9f49cddaf2f0505b2770d385dfeefe1"} Nov 03 20:46:56.106825 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:56.106439 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j5mgz" podStartSLOduration=2.7908517330000002 podStartE2EDuration="8.106418629s" totalImagesPullingTime="5.315566896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:56.105226116 +0000 UTC m=+8.706219172" watchObservedRunningTime="2026-11-03 20:46:56.106418629 +0000 UTC m=+8.707411683" Nov 03 20:46:56.739718 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:56.739677 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:46:56.739879 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:56.739855 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:46:56.739974 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:56.739916 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret podName:f1eb3e05-8b22-4992-97d9-c71ecc6b99dc nodeName:}" failed. No retries permitted until 2026-11-03 20:47:04.739898497 +0000 UTC m=+17.340891537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret") pod "global-pull-secret-syncer-lvbhk" (UID: "f1eb3e05-8b22-4992-97d9-c71ecc6b99dc") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:46:56.840725 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:56.840682 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:46:56.840919 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:56.840891 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:46:56.841003 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:56.840982 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs podName:84556453-e5d9-4449-ba59-819c248e1e2a nodeName:}" failed. No retries permitted until 2026-11-03 20:47:04.840944634 +0000 UTC m=+17.441937680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs") pod "network-metrics-daemon-mr55c" (UID: "84556453-e5d9-4449-ba59-819c248e1e2a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:46:56.900864 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:56.900290 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:46:56.900864 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:56.900419 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7lxn" podUID="af51f192-4170-4d3b-85ae-7d2ceb14de05" Nov 03 20:46:56.900864 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:56.900498 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:46:56.900864 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:56.900577 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lvbhk" podUID="f1eb3e05-8b22-4992-97d9-c71ecc6b99dc" Nov 03 20:46:56.900864 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:56.900638 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:46:56.900864 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:56.900721 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr55c" podUID="84556453-e5d9-4449-ba59-819c248e1e2a" Nov 03 20:46:56.941996 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:56.941665 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd729\" (UniqueName: \"kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729\") pod \"network-check-target-l7lxn\" (UID: \"af51f192-4170-4d3b-85ae-7d2ceb14de05\") " pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:46:56.941996 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:56.941840 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:46:56.941996 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:56.941859 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:46:56.941996 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:56.941872 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rd729 for pod openshift-network-diagnostics/network-check-target-l7lxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:46:56.941996 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:56.941934 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729 podName:af51f192-4170-4d3b-85ae-7d2ceb14de05 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:04.941914374 +0000 UTC m=+17.542907415 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd729" (UniqueName: "kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729") pod "network-check-target-l7lxn" (UID: "af51f192-4170-4d3b-85ae-7d2ceb14de05") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:46:58.003999 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:58.003934 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r9lld" event={"ID":"462e871a-1226-4f07-b34c-ef612fa6bf52","Type":"ContainerStarted","Data":"71958822528fc406b1592536adbfa4fd60244075090a5b6ca86258b71d607e3b"} Nov 03 20:46:58.008827 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:58.008737 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" event={"ID":"245acdd3-cebd-49fe-bf1e-867df120ac8e","Type":"ContainerStarted","Data":"ae8718a3c9fe755617eef642f01b05485eb29f8c28ccc774296392e832d6b781"} Nov 03 20:46:58.131762 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:58.131611 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" podStartSLOduration=9.13158745 podStartE2EDuration="9.13158745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:56.234644243 +0000 UTC m=+8.835637291" watchObservedRunningTime="2026-11-03 20:46:58.13158745 +0000 UTC m=+10.732580498" Nov 03 20:46:58.132370 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:58.132326 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r9lld" podStartSLOduration=2.8036996199999997 podStartE2EDuration="9.132314664s" totalImagesPullingTime="6.328615044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:58.131137093 +0000 UTC m=+10.732130142" watchObservedRunningTime="2026-11-03 20:46:58.132314664 +0000 UTC m=+10.733307715" Nov 03 20:46:58.900750 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:58.900709 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:46:58.900952 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:58.900846 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7lxn" podUID="af51f192-4170-4d3b-85ae-7d2ceb14de05" Nov 03 20:46:58.901252 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:58.901231 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:46:58.901368 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:58.901343 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lvbhk" podUID="f1eb3e05-8b22-4992-97d9-c71ecc6b99dc" Nov 03 20:46:58.901443 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:58.901418 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:46:58.901529 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:46:58.901509 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr55c" podUID="84556453-e5d9-4449-ba59-819c248e1e2a" Nov 03 20:46:59.826795 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:59.826762 2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 20:46:59.843947 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:59.843200 2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:46:59.826787534Z","UUID":"adaa12b2-25c0-45b0-aa4a-bded58cd6dd9","Handler":null,"Name":"","Endpoint":""} Nov 03 20:46:59.844915 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:59.844890 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 20:46:59.845048 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:46:59.844947 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 20:47:00.016809 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:00.016765 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" event={"ID":"245acdd3-cebd-49fe-bf1e-867df120ac8e","Type":"ContainerStarted","Data":"19ab2c3e26b0458546198a1a324c3d9290e64b5ce1bf0fb1e6fc2f865c47b51d"} Nov 03 20:47:00.018871 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:00.018837 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-snszh" event={"ID":"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20","Type":"ContainerStarted","Data":"95d91db544f62195a7fb8f011377b9d8b0eaa36f306a7e5169b11d1cf63236dd"} Nov 03 20:47:00.021847 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:00.021818 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sm2mn" event={"ID":"c7559e5a-a8fd-450c-8f40-f2072e93a9b5","Type":"ContainerStarted","Data":"11e54b65f47401074a887e1cb8235ffadbabb8de348dbf17a4c66375968b4ee1"} Nov 03 20:47:00.279872 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:00.279802 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-snszh" podStartSLOduration=3.204763542 podStartE2EDuration="12.279779296s" totalImagesPullingTime="9.075015754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:00.151301278 +0000 UTC m=+12.752294326" watchObservedRunningTime="2026-11-03 20:47:00.279779296 +0000 UTC m=+12.880772347" Nov 03 20:47:00.485225 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:00.485194 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j5mgz" Nov 03 20:47:00.485720 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:00.485697 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j5mgz" Nov 03 20:47:00.605525 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:00.605427 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-sm2mn" podStartSLOduration=2.33309626 podStartE2EDuration="11.605404371s" totalImagesPullingTime="9.272308111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:00.280429891 +0000 UTC m=+12.881422939" watchObservedRunningTime="2026-11-03 20:47:00.605404371 +0000 UTC m=+13.206397416" Nov 03 20:47:00.899741 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:00.899709 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:00.900151 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:00.899846 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lvbhk" podUID="f1eb3e05-8b22-4992-97d9-c71ecc6b99dc" Nov 03 20:47:00.900151 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:00.900001 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:00.900151 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:00.900001 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:00.900151 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:00.900121 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr55c" podUID="84556453-e5d9-4449-ba59-819c248e1e2a" Nov 03 20:47:00.900275 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:00.900159 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7lxn" podUID="af51f192-4170-4d3b-85ae-7d2ceb14de05" Nov 03 20:47:01.026800 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:01.026755 2583 generic.go:356] "Generic (PLEG): container finished" podID="fb02f36e-58c6-47fb-b6f7-0ba78d029ffb" containerID="564fbee1f50e6deca900e61bc9758d4dad87f126e9bdf969802345680218971d" exitCode=0 Nov 03 20:47:01.027106 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:01.027063 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tgl9l" event={"ID":"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb","Type":"ContainerDied","Data":"564fbee1f50e6deca900e61bc9758d4dad87f126e9bdf969802345680218971d"} Nov 03 20:47:02.031534 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:02.031310 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" event={"ID":"245acdd3-cebd-49fe-bf1e-867df120ac8e","Type":"ContainerStarted","Data":"b8414661f69bc608f87e75da01d96c6027f6a088f911b120eb7658296e391a12"} Nov 03 20:47:02.033135 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:02.033048 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" event={"ID":"eafb0c22-9550-4b1c-8134-fc80562025eb","Type":"ContainerStarted","Data":"ba4b23e20ddd6d92c8f8e35e784c34a517704d092b6c03d39dd75fdd360d2427"} Nov 03 20:47:02.034622 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:02.034601 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wl57x" event={"ID":"542dd56a-cfab-40e2-a01b-a172651d46f8","Type":"ContainerStarted","Data":"9e48c09653aa191997ba72ced4063829c373372bdcd45acf1cf02d90e829b150"} Nov 03 20:47:02.152783 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:02.152691 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" podStartSLOduration=2.389031609 podStartE2EDuration="13.152670279s" totalImagesPullingTime="10.76363867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:46:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:02.152547338 +0000 UTC m=+14.753540382" watchObservedRunningTime="2026-11-03 20:47:02.152670279 +0000 UTC m=+14.753663325" Nov 03 20:47:02.276543 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:02.276486 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" podStartSLOduration=2.2743042669999998 podStartE2EDuration="13.276415251s" totalImagesPullingTime="11.002110984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:02.275748829 +0000 UTC m=+14.876741873" watchObservedRunningTime="2026-11-03 20:47:02.276415251 +0000 UTC m=+14.877408295" Nov 03 20:47:02.393254 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:02.392682 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wl57x" podStartSLOduration=3.870717156 podStartE2EDuration="13.39266205s" totalImagesPullingTime="9.521944894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:02.392378359 +0000 UTC m=+14.993371404" watchObservedRunningTime="2026-11-03 20:47:02.39266205 +0000 UTC m=+14.993655096" Nov 03 20:47:02.900364 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:02.900277 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:02.900537 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:02.900299 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:02.900537 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:02.900410 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7lxn" podUID="af51f192-4170-4d3b-85ae-7d2ceb14de05" Nov 03 20:47:02.900537 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:02.900416 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:02.900537 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:02.900496 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lvbhk" podUID="f1eb3e05-8b22-4992-97d9-c71ecc6b99dc" Nov 03 20:47:02.900718 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:02.900594 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr55c" podUID="84556453-e5d9-4449-ba59-819c248e1e2a" Nov 03 20:47:03.039080 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:03.039046 2583 generic.go:356] "Generic (PLEG): container finished" podID="fb02f36e-58c6-47fb-b6f7-0ba78d029ffb" containerID="d064963297fba07484bd3b9d45b5d0f2580f6b14892fdec69bb91cbb35989123" exitCode=0 Nov 03 20:47:03.039661 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:03.039097 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tgl9l" event={"ID":"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb","Type":"ContainerDied","Data":"d064963297fba07484bd3b9d45b5d0f2580f6b14892fdec69bb91cbb35989123"} Nov 03 20:47:03.359847 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:03.359759 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j5mgz" Nov 03 20:47:03.360010 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:03.359976 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:47:03.360398 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:03.360374 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j5mgz" Nov 03 20:47:04.806353 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:04.806261 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:04.807189 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:04.806436 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:47:04.807189 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:04.806505 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret podName:f1eb3e05-8b22-4992-97d9-c71ecc6b99dc nodeName:}" failed. No retries permitted until 2026-11-03 20:47:20.806487954 +0000 UTC m=+33.407480993 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret") pod "global-pull-secret-syncer-lvbhk" (UID: "f1eb3e05-8b22-4992-97d9-c71ecc6b99dc") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:47:04.900234 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:04.900196 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:04.900424 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:04.900195 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:04.900424 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:04.900329 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr55c" podUID="84556453-e5d9-4449-ba59-819c248e1e2a" Nov 03 20:47:04.900424 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:04.900195 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:04.900553 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:04.900405 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7lxn" podUID="af51f192-4170-4d3b-85ae-7d2ceb14de05" Nov 03 20:47:04.900553 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:04.900509 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lvbhk" podUID="f1eb3e05-8b22-4992-97d9-c71ecc6b99dc" Nov 03 20:47:04.907436 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:04.907392 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:04.907600 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:04.907537 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:47:04.907648 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:04.907614 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs podName:84556453-e5d9-4449-ba59-819c248e1e2a nodeName:}" failed. No retries permitted until 2026-11-03 20:47:20.907598113 +0000 UTC m=+33.508591136 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs") pod "network-metrics-daemon-mr55c" (UID: "84556453-e5d9-4449-ba59-819c248e1e2a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:47:05.007797 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:05.007761 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd729\" (UniqueName: \"kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729\") pod \"network-check-target-l7lxn\" (UID: \"af51f192-4170-4d3b-85ae-7d2ceb14de05\") " pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:05.007969 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:05.007918 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:47:05.007969 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:05.007937 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:47:05.007969 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:05.007946 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rd729 for pod openshift-network-diagnostics/network-check-target-l7lxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:47:05.008094 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:05.008001 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729 podName:af51f192-4170-4d3b-85ae-7d2ceb14de05 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:21.007983057 +0000 UTC m=+33.608976081 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd729" (UniqueName: "kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729") pod "network-check-target-l7lxn" (UID: "af51f192-4170-4d3b-85ae-7d2ceb14de05") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:47:05.046704 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:05.046659 2583 generic.go:356] "Generic (PLEG): container finished" podID="fb02f36e-58c6-47fb-b6f7-0ba78d029ffb" containerID="31673d96c7942ccf8c163b6db8df1a65fcf6d62c34f82359766527b25d308257" exitCode=0 Nov 03 20:47:05.046865 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:05.046716 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tgl9l" event={"ID":"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb","Type":"ContainerDied","Data":"31673d96c7942ccf8c163b6db8df1a65fcf6d62c34f82359766527b25d308257"} Nov 03 20:47:06.900095 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:06.899825 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:06.900623 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:06.899860 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:06.900623 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:06.900154 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lvbhk" podUID="f1eb3e05-8b22-4992-97d9-c71ecc6b99dc" Nov 03 20:47:06.900623 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:06.899860 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:06.900623 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:06.900262 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr55c" podUID="84556453-e5d9-4449-ba59-819c248e1e2a" Nov 03 20:47:06.900623 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:06.900352 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7lxn" podUID="af51f192-4170-4d3b-85ae-7d2ceb14de05" Nov 03 20:47:07.055652 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:07.055451 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" event={"ID":"8e456784-7491-426a-80e5-17a9dfa2f07f","Type":"ContainerStarted","Data":"285dc16346a3cce443299ef3c683b3bba3a447a447a072ea41e4ace97e7e6a40"} Nov 03 20:47:07.055821 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:07.055713 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" event={"ID":"8e456784-7491-426a-80e5-17a9dfa2f07f","Type":"ContainerStarted","Data":"340399efbbca4381406df3294eb94cac7ea27c7b7ddac04b813c3a80cbd6af19"} Nov 03 20:47:07.055821 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:07.055761 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" event={"ID":"8e456784-7491-426a-80e5-17a9dfa2f07f","Type":"ContainerStarted","Data":"34b6379cd80b39051a2c3ea4bb759f322faad88ef47f4c8c1d57dc5b40279eb5"} Nov 03 20:47:07.055821 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:07.055789 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" event={"ID":"8e456784-7491-426a-80e5-17a9dfa2f07f","Type":"ContainerStarted","Data":"bf643beb3f563c5679cbdc326ae3199fb550885d982931ee058c59f73a932129"} Nov 03 20:47:07.055988 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:07.055821 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" event={"ID":"8e456784-7491-426a-80e5-17a9dfa2f07f","Type":"ContainerStarted","Data":"edb38202a4fc9724f13eac09c3cb76be5a4e008f5a9024eb475b9dcb6d7d1374"} Nov 03 20:47:07.055988 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:07.055852 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" event={"ID":"8e456784-7491-426a-80e5-17a9dfa2f07f","Type":"ContainerStarted","Data":"e12c3f6aedac6e3d7b0d3ef85458feaae23264b2bcf100995ff29d5bd7766816"} Nov 03 20:47:08.060582 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:08.060551 2583 generic.go:356] "Generic (PLEG): container finished" podID="fb02f36e-58c6-47fb-b6f7-0ba78d029ffb" containerID="ae6a52571df0bd4f4777619ad703b9db45e3162a9cd9f36e960f6916cf71c0d4" exitCode=0 Nov 03 20:47:08.061060 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:08.060603 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tgl9l" event={"ID":"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb","Type":"ContainerDied","Data":"ae6a52571df0bd4f4777619ad703b9db45e3162a9cd9f36e960f6916cf71c0d4"} Nov 03 20:47:08.899994 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:08.899956 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:08.900140 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:08.899970 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:08.900140 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:08.900061 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lvbhk" podUID="f1eb3e05-8b22-4992-97d9-c71ecc6b99dc" Nov 03 20:47:08.900240 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:08.900145 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr55c" podUID="84556453-e5d9-4449-ba59-819c248e1e2a" Nov 03 20:47:08.900240 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:08.899977 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:08.900240 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:08.900223 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7lxn" podUID="af51f192-4170-4d3b-85ae-7d2ceb14de05" Nov 03 20:47:09.066131 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:09.066039 2583 generic.go:356] "Generic (PLEG): container finished" podID="fb02f36e-58c6-47fb-b6f7-0ba78d029ffb" containerID="30a04279adaae404f1d16cc19a01b2420f311187ddfadc4e43852d6f83c312f8" exitCode=0 Nov 03 20:47:09.066688 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:09.066125 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tgl9l" event={"ID":"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb","Type":"ContainerDied","Data":"30a04279adaae404f1d16cc19a01b2420f311187ddfadc4e43852d6f83c312f8"} Nov 03 20:47:09.069514 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:09.069483 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" event={"ID":"8e456784-7491-426a-80e5-17a9dfa2f07f","Type":"ContainerStarted","Data":"20c887661ea4afd31be1f0a4f5d2334534d3bf02677df85c303546595874ed91"} Nov 03 20:47:10.074159 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:10.074130 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tgl9l" event={"ID":"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb","Type":"ContainerStarted","Data":"9ff3c3c66cf14613e882a630c33555eda67e9161a0f1c062ed754e57060fb337"} Nov 03 20:47:10.301159 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:10.301108 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tgl9l" podStartSLOduration=7.103125837 podStartE2EDuration="21.301091018s" totalImagesPullingTime="14.197965181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:46:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:10.249839289 +0000 UTC m=+22.850832345" watchObservedRunningTime="2026-11-03 20:47:10.301091018 +0000 UTC m=+22.902084058" Nov 03 20:47:10.899861 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:10.899827 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:10.899861 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:10.899851 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:10.900114 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:10.899858 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:10.900114 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:10.899971 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7lxn" podUID="af51f192-4170-4d3b-85ae-7d2ceb14de05" Nov 03 20:47:10.900114 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:10.900058 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr55c" podUID="84556453-e5d9-4449-ba59-819c248e1e2a" Nov 03 20:47:10.900114 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:10.900105 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lvbhk" podUID="f1eb3e05-8b22-4992-97d9-c71ecc6b99dc" Nov 03 20:47:11.080189 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:11.079852 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" event={"ID":"8e456784-7491-426a-80e5-17a9dfa2f07f","Type":"ContainerStarted","Data":"f46cefbd258c6a1d422eb3b49899ad8f8236d603bfd43266c2e5e2f4f0ba8f9b"} Nov 03 20:47:11.320692 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:11.320629 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" podStartSLOduration=6.697353681 podStartE2EDuration="22.320616031s" totalImagesPullingTime="15.62326235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:11.320119142 +0000 UTC m=+23.921112187" watchObservedRunningTime="2026-11-03 20:47:11.320616031 +0000 UTC m=+23.921609075" Nov 03 20:47:12.090601 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:12.089469 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:12.090601 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:12.089654 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:12.090601 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:12.089717 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:12.114182 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:12.114153 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:12.115645 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:12.115625 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:12.899940 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:12.899894 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:12.900094 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:12.899966 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:12.900094 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:12.900059 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:12.900094 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:12.900065 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7lxn" podUID="af51f192-4170-4d3b-85ae-7d2ceb14de05" Nov 03 20:47:12.900186 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:12.900142 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr55c" podUID="84556453-e5d9-4449-ba59-819c248e1e2a" Nov 03 20:47:12.900223 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:12.900195 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lvbhk" podUID="f1eb3e05-8b22-4992-97d9-c71ecc6b99dc" Nov 03 20:47:13.553850 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:13.553815 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lvbhk"] Nov 03 20:47:13.554567 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:13.554548 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:13.554822 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:13.554798 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lvbhk" podUID="f1eb3e05-8b22-4992-97d9-c71ecc6b99dc" Nov 03 20:47:13.556511 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:13.556485 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l7lxn"] Nov 03 20:47:13.556622 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:13.556577 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:13.556673 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:13.556647 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7lxn" podUID="af51f192-4170-4d3b-85ae-7d2ceb14de05" Nov 03 20:47:13.558791 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:13.558770 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mr55c"] Nov 03 20:47:13.558872 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:13.558850 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:13.558953 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:13.558937 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr55c" podUID="84556453-e5d9-4449-ba59-819c248e1e2a" Nov 03 20:47:14.900549 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:14.900516 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:14.900997 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:14.900628 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7lxn" podUID="af51f192-4170-4d3b-85ae-7d2ceb14de05" Nov 03 20:47:15.899944 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:15.899846 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:15.900100 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:15.899847 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:15.900100 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:15.899981 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr55c" podUID="84556453-e5d9-4449-ba59-819c248e1e2a" Nov 03 20:47:15.900100 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:15.900047 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lvbhk" podUID="f1eb3e05-8b22-4992-97d9-c71ecc6b99dc" Nov 03 20:47:16.900185 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:16.900098 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:16.900576 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:16.900209 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7lxn" podUID="af51f192-4170-4d3b-85ae-7d2ceb14de05" Nov 03 20:47:17.900811 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:17.900786 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:17.901188 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:17.900928 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:17.901188 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:17.900927 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lvbhk" podUID="f1eb3e05-8b22-4992-97d9-c71ecc6b99dc" Nov 03 20:47:17.901188 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:17.901084 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr55c" podUID="84556453-e5d9-4449-ba59-819c248e1e2a" Nov 03 20:47:18.900071 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:18.900041 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:18.900222 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:18.900144 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7lxn" podUID="af51f192-4170-4d3b-85ae-7d2ceb14de05" Nov 03 20:47:19.677467 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.677384 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeReady" Nov 03 20:47:19.677841 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.677555 2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 20:47:19.837263 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.837231 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kwzxh"] Nov 03 20:47:19.856529 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.856497 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ghxn6"] Nov 03 20:47:19.856680 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.856615 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kwzxh" Nov 03 20:47:19.872852 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.872651 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bd5l5"] Nov 03 20:47:19.872981 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.872760 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ghxn6" Nov 03 20:47:19.886863 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.886833 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk"] Nov 03 20:47:19.887032 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.886985 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bd5l5" Nov 03 20:47:19.908984 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.908956 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:19.911279 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.911242 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkv6p\" (UniqueName: \"kubernetes.io/projected/5ea4f6f2-4699-46d7-91bc-8391dde191eb-kube-api-access-gkv6p\") pod \"kas-connection-checker-685895c7c9-kwzxh\" (UID: \"5ea4f6f2-4699-46d7-91bc-8391dde191eb\") " pod="kube-system/kas-connection-checker-685895c7c9-kwzxh" Nov 03 20:47:19.913206 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.913180 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-92hts"] Nov 03 20:47:19.913328 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.913296 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:19.913328 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.913309 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:19.913446 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.913387 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:19.938676 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.938652 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-ccxnp"] Nov 03 20:47:19.938783 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.938771 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:19.941512 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.940739 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 20:47:19.941512 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.940770 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 20:47:19.956206 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.956045 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7fb544c95-8798s"] Nov 03 20:47:19.956206 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.956117 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:19.968651 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.963399 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 20:47:19.968651 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.968528 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-d5cfx\"" Nov 03 20:47:19.974920 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.974885 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp"] Nov 03 20:47:19.975069 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.975020 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:19.995696 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.995660 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc"] Nov 03 20:47:19.995860 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:19.995737 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:20.011655 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.011622 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:20.011813 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.011666 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d21a1ae-f712-4f89-b34f-8016fd918484-service-ca-bundle\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.011813 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.011697 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkv6p\" (UniqueName: \"kubernetes.io/projected/5ea4f6f2-4699-46d7-91bc-8391dde191eb-kube-api-access-gkv6p\") pod \"kas-connection-checker-685895c7c9-kwzxh\" (UID: \"5ea4f6f2-4699-46d7-91bc-8391dde191eb\") " pod="kube-system/kas-connection-checker-685895c7c9-kwzxh" Nov 03 20:47:20.011813 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.011748 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d21a1ae-f712-4f89-b34f-8016fd918484-tmp\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.011989 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.011919 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d21a1ae-f712-4f89-b34f-8016fd918484-serving-cert\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.011989 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.011950 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/24a97420-efcd-4a6f-b072-1e5792264188-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:20.012100 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.012008 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9dm2\" (UniqueName: \"kubernetes.io/projected/e863782c-c289-4fa5-93b1-b475358a5001-kube-api-access-l9dm2\") pod \"kas-connection-checker-685895c7c9-bd5l5\" (UID: \"e863782c-c289-4fa5-93b1-b475358a5001\") " pod="kube-system/kas-connection-checker-685895c7c9-bd5l5" Nov 03 20:47:20.012100 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.012036 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d21a1ae-f712-4f89-b34f-8016fd918484-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.012100 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.012076 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3c555a1-e6bd-4042-be58-404cb6693922-trusted-ca\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:20.012258 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.012146 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3c555a1-e6bd-4042-be58-404cb6693922-serving-cert\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:20.012258 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.012180 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8d21a1ae-f712-4f89-b34f-8016fd918484-snapshots\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.012351 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.012254 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zwts\" (UniqueName: \"kubernetes.io/projected/34ba944d-b672-489d-9acc-fde356202fbe-kube-api-access-6zwts\") pod \"kas-connection-checker-685895c7c9-ghxn6\" (UID: \"34ba944d-b672-489d-9acc-fde356202fbe\") " pod="kube-system/kas-connection-checker-685895c7c9-ghxn6" Nov 03 20:47:20.012351 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.012290 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3c555a1-e6bd-4042-be58-404cb6693922-config\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:20.012423 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.012369 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zvs2\" (UniqueName: \"kubernetes.io/projected/24a97420-efcd-4a6f-b072-1e5792264188-kube-api-access-8zvs2\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:20.012423 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.012400 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltgln\" (UniqueName: \"kubernetes.io/projected/e3c555a1-e6bd-4042-be58-404cb6693922-kube-api-access-ltgln\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:20.012423 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.012419 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hd42\" (UniqueName: \"kubernetes.io/projected/8d21a1ae-f712-4f89-b34f-8016fd918484-kube-api-access-2hd42\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.014914 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.014883 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4"] Nov 03 20:47:20.015018 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.014982 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" Nov 03 20:47:20.031886 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.031853 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nm757"] Nov 03 20:47:20.031886 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.031873 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" Nov 03 20:47:20.041065 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.041031 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:47:20.041065 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.041064 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 20:47:20.041290 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.041112 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 20:47:20.041290 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.041143 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 20:47:20.041290 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.041142 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 20:47:20.041290 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.041159 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:47:20.041290 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.041226 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:47:20.041290 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.041233 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 20:47:20.041290 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.041241 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 20:47:20.041290 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.041264 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 20:47:20.041290 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.041268 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:47:20.041290 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.041273 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:47:20.041805 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.041529 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 20:47:20.048401 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.048374 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75999699c7-pfc8g"] Nov 03 20:47:20.048543 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.048516 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nm757" Nov 03 20:47:20.072497 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.072460 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-xwt2z"] Nov 03 20:47:20.072650 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.072597 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.076771 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.076747 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-687954fc7b-25pdn"] Nov 03 20:47:20.076921 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.076844 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:20.080555 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.080539 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 20:47:20.086994 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.086974 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-t4tng"] Nov 03 20:47:20.087123 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.087109 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.093530 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.093504 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:47:20.093631 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.093595 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 20:47:20.093631 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.093610 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 20:47:20.093722 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.093631 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 20:47:20.093722 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.093662 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 20:47:20.093789 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.093754 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:47:20.093789 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.093766 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 20:47:20.093861 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.093791 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 20:47:20.093861 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.093837 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-p57f6\"" Nov 03 20:47:20.093861 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.093853 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l9gsz\"" Nov 03 20:47:20.094006 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.093872 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lw8vn\"" Nov 03 20:47:20.094006 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.093887 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 20:47:20.094006 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.093891 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 20:47:20.094006 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.093925 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 20:47:20.094006 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.093939 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 20:47:20.094416 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.094371 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-g5tq4\"" Nov 03 20:47:20.094688 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.094666 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:47:20.094990 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.094971 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-92gvj\"" Nov 03 20:47:20.095280 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.095262 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-p5wrg\"" Nov 03 20:47:20.095372 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.095351 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9c99h"] Nov 03 20:47:20.095425 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.095415 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 20:47:20.095749 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.095730 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-snm99\"" Nov 03 20:47:20.096642 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.096620 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-78pt4\"" Nov 03 20:47:20.096991 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.096973 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 20:47:20.097931 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.097282 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-964vd\"" Nov 03 20:47:20.097931 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.097688 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4tng" Nov 03 20:47:20.100385 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.100363 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkv6p\" (UniqueName: \"kubernetes.io/projected/5ea4f6f2-4699-46d7-91bc-8391dde191eb-kube-api-access-gkv6p\") pod \"kas-connection-checker-685895c7c9-kwzxh\" (UID: \"5ea4f6f2-4699-46d7-91bc-8391dde191eb\") " pod="kube-system/kas-connection-checker-685895c7c9-kwzxh" Nov 03 20:47:20.105319 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.105298 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-njb7z"] Nov 03 20:47:20.105442 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.105426 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9c99h" Nov 03 20:47:20.111797 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.111779 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kwzxh"] Nov 03 20:47:20.111888 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.111810 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bd5l5"] Nov 03 20:47:20.111888 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.111824 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nm757"] Nov 03 20:47:20.111888 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.111837 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk"] Nov 03 20:47:20.112010 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.111928 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4"] Nov 03 20:47:20.112010 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.111937 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:20.112094 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.111943 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp"] Nov 03 20:47:20.112094 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.112037 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-92hts"] Nov 03 20:47:20.112094 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.112054 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75999699c7-pfc8g"] Nov 03 20:47:20.112094 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.112069 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ghxn6"] Nov 03 20:47:20.112094 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.112083 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc"] Nov 03 20:47:20.112265 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.112097 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-687954fc7b-25pdn"] Nov 03 20:47:20.112265 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.112116 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-njb7z"] Nov 03 20:47:20.112265 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.112144 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-xwt2z"] Nov 03 20:47:20.112265 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.112161 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-t4tng"] Nov 03 20:47:20.112265 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.112175 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-ccxnp"] Nov 03 20:47:20.112265 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.112188 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7fb544c95-8798s"] Nov 03 20:47:20.112265 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.112210 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9c99h"] Nov 03 20:47:20.112868 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.112844 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvs2\" (UniqueName: \"kubernetes.io/projected/24a97420-efcd-4a6f-b072-1e5792264188-kube-api-access-8zvs2\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:20.112971 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.112886 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltgln\" (UniqueName: \"kubernetes.io/projected/e3c555a1-e6bd-4042-be58-404cb6693922-kube-api-access-ltgln\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:20.113021 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.112975 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:20.113021 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113004 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d21a1ae-f712-4f89-b34f-8016fd918484-service-ca-bundle\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.113117 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113036 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj7td\" (UniqueName: \"kubernetes.io/projected/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-kube-api-access-sj7td\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:20.113117 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113101 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d21a1ae-f712-4f89-b34f-8016fd918484-tmp\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.113211 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113177 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mzl9\" (UniqueName: \"kubernetes.io/projected/e29718e2-f954-4ec8-9db2-d6c3638bc51c-kube-api-access-6mzl9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4\" (UID: \"e29718e2-f954-4ec8-9db2-d6c3638bc51c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" Nov 03 20:47:20.113263 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113207 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-stats-auth\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:20.113263 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113231 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/510fbc36-62de-4096-b8e8-d8606a4ad7d5-config\") pod \"service-ca-operator-544d749c8c-s82vc\" (UID: \"510fbc36-62de-4096-b8e8-d8606a4ad7d5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" Nov 03 20:47:20.113366 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113267 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e29718e2-f954-4ec8-9db2-d6c3638bc51c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4\" (UID: \"e29718e2-f954-4ec8-9db2-d6c3638bc51c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" Nov 03 20:47:20.113366 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113296 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhh48\" (UniqueName: \"kubernetes.io/projected/510fbc36-62de-4096-b8e8-d8606a4ad7d5-kube-api-access-fhh48\") pod \"service-ca-operator-544d749c8c-s82vc\" (UID: \"510fbc36-62de-4096-b8e8-d8606a4ad7d5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" Nov 03 20:47:20.113366 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113322 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cjnm\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-kube-api-access-5cjnm\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.113366 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113349 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d21a1ae-f712-4f89-b34f-8016fd918484-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.113574 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113383 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-default-certificate\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:20.113574 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113421 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3c555a1-e6bd-4042-be58-404cb6693922-trusted-ca\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:20.113574 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113427 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d21a1ae-f712-4f89-b34f-8016fd918484-tmp\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.113574 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.113109 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:47:20.113574 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113450 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nd4m\" (UniqueName: \"kubernetes.io/projected/8f39698b-0204-4492-ac16-e1728e71b8d6-kube-api-access-8nd4m\") pod \"volume-data-source-validator-58c9f5f46d-nm757\" (UID: \"8f39698b-0204-4492-ac16-e1728e71b8d6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nm757" Nov 03 20:47:20.113574 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113480 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3c555a1-e6bd-4042-be58-404cb6693922-serving-cert\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:20.113574 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.113513 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls podName:24a97420-efcd-4a6f-b072-1e5792264188 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:20.613495131 +0000 UTC m=+33.214488162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gkklk" (UID: "24a97420-efcd-4a6f-b072-1e5792264188") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:47:20.113574 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113532 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/510fbc36-62de-4096-b8e8-d8606a4ad7d5-serving-cert\") pod \"service-ca-operator-544d749c8c-s82vc\" (UID: \"510fbc36-62de-4096-b8e8-d8606a4ad7d5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" Nov 03 20:47:20.113997 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113793 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/793c02dd-468b-4308-bac2-67d04e40579c-ca-trust-extracted\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.113997 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113823 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-installation-pull-secrets\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.113997 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113929 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zwts\" (UniqueName: \"kubernetes.io/projected/34ba944d-b672-489d-9acc-fde356202fbe-kube-api-access-6zwts\") pod \"kas-connection-checker-685895c7c9-ghxn6\" (UID: \"34ba944d-b672-489d-9acc-fde356202fbe\") " pod="kube-system/kas-connection-checker-685895c7c9-ghxn6" Nov 03 20:47:20.113997 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.113974 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3c555a1-e6bd-4042-be58-404cb6693922-config\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:20.114179 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.114083 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hd42\" (UniqueName: \"kubernetes.io/projected/8d21a1ae-f712-4f89-b34f-8016fd918484-kube-api-access-2hd42\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.114179 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.114114 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:20.114531 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.114502 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3c555a1-e6bd-4042-be58-404cb6693922-trusted-ca\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:20.114624 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.114593 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-registry-certificates\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.114624 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.114595 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3c555a1-e6bd-4042-be58-404cb6693922-config\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:20.114740 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.114703 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d21a1ae-f712-4f89-b34f-8016fd918484-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.114740 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.114703 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d21a1ae-f712-4f89-b34f-8016fd918484-serving-cert\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.114850 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.114768 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/24a97420-efcd-4a6f-b072-1e5792264188-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:20.114850 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.114837 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m4g8\" (UniqueName: \"kubernetes.io/projected/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-kube-api-access-5m4g8\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:20.115005 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.114979 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-image-registry-private-configuration\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.115065 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.115019 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-trusted-ca\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.115122 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.115069 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9dm2\" (UniqueName: \"kubernetes.io/projected/e863782c-c289-4fa5-93b1-b475358a5001-kube-api-access-l9dm2\") pod \"kas-connection-checker-685895c7c9-bd5l5\" (UID: \"e863782c-c289-4fa5-93b1-b475358a5001\") " pod="kube-system/kas-connection-checker-685895c7c9-bd5l5" Nov 03 20:47:20.115122 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.115096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:20.115233 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.115124 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.115233 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.115151 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-bound-sa-token\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.115233 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.115178 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:20.115233 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.115206 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e29718e2-f954-4ec8-9db2-d6c3638bc51c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4\" (UID: \"e29718e2-f954-4ec8-9db2-d6c3638bc51c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" Nov 03 20:47:20.115427 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.115244 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8d21a1ae-f712-4f89-b34f-8016fd918484-snapshots\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.115427 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.115269 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:20.115576 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.115555 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d21a1ae-f712-4f89-b34f-8016fd918484-service-ca-bundle\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.115633 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.115615 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/24a97420-efcd-4a6f-b072-1e5792264188-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:20.116239 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.116216 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8d21a1ae-f712-4f89-b34f-8016fd918484-snapshots\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.117345 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.117322 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3c555a1-e6bd-4042-be58-404cb6693922-serving-cert\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:20.117584 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.117560 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d21a1ae-f712-4f89-b34f-8016fd918484-serving-cert\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.141611 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.141583 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 20:47:20.141750 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.141628 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:47:20.141750 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.141650 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 20:47:20.141750 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.141628 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:47:20.141750 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.141673 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 20:47:20.141750 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.141653 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:47:20.141750 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.141694 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 20:47:20.141965 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.141769 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 20:47:20.142005 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.141972 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 20:47:20.143812 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.143794 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 20:47:20.158652 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.158632 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:47:20.158795 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.158776 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:47:20.164289 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.164264 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 20:47:20.164428 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.164414 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 20:47:20.164509 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.164496 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:47:20.164566 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.164514 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 20:47:20.164566 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.164544 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-99tc2\"" Nov 03 20:47:20.164566 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.164555 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d7jn9\"" Nov 03 20:47:20.164701 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.164581 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 20:47:20.164701 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.164585 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j76n8\"" Nov 03 20:47:20.164701 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.164593 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-s2xmz\"" Nov 03 20:47:20.164701 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.164652 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rf5zr\"" Nov 03 20:47:20.165045 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.165030 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:47:20.165233 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.165222 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:47:20.165274 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.165251 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-68sfv\"" Nov 03 20:47:20.176317 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.176284 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvs2\" (UniqueName: \"kubernetes.io/projected/24a97420-efcd-4a6f-b072-1e5792264188-kube-api-access-8zvs2\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:20.176453 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.176359 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltgln\" (UniqueName: \"kubernetes.io/projected/e3c555a1-e6bd-4042-be58-404cb6693922-kube-api-access-ltgln\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:20.176624 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.176602 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zwts\" (UniqueName: \"kubernetes.io/projected/34ba944d-b672-489d-9acc-fde356202fbe-kube-api-access-6zwts\") pod \"kas-connection-checker-685895c7c9-ghxn6\" (UID: \"34ba944d-b672-489d-9acc-fde356202fbe\") " pod="kube-system/kas-connection-checker-685895c7c9-ghxn6" Nov 03 20:47:20.176676 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.176654 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hd42\" (UniqueName: \"kubernetes.io/projected/8d21a1ae-f712-4f89-b34f-8016fd918484-kube-api-access-2hd42\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.176943 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.176928 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9dm2\" (UniqueName: \"kubernetes.io/projected/e863782c-c289-4fa5-93b1-b475358a5001-kube-api-access-l9dm2\") pod \"kas-connection-checker-685895c7c9-bd5l5\" (UID: \"e863782c-c289-4fa5-93b1-b475358a5001\") " pod="kube-system/kas-connection-checker-685895c7c9-bd5l5" Nov 03 20:47:20.178832 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.178818 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kwzxh" Nov 03 20:47:20.199553 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.199525 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ghxn6" Nov 03 20:47:20.216471 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.216442 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7262fd-b29d-4c72-b717-ef9742bdd2db-config-volume\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:20.216585 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.216481 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.216585 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.216508 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m4g8\" (UniqueName: \"kubernetes.io/projected/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-kube-api-access-5m4g8\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:20.216696 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.216658 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-image-registry-private-configuration\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.216696 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.216680 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-trusted-ca\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.216796 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.216704 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fflvq\" (UniqueName: \"kubernetes.io/projected/42d01b00-5ec9-4f8d-a8f2-10e0addd1b67-kube-api-access-fflvq\") pod \"ingress-canary-9c99h\" (UID: \"42d01b00-5ec9-4f8d-a8f2-10e0addd1b67\") " pod="openshift-ingress-canary/ingress-canary-9c99h" Nov 03 20:47:20.216796 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.216758 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:20.216992 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.216953 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-bound-sa-token\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.217097 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.217002 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.217097 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.217025 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-bound-sa-token\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.217214 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.217098 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:20.217214 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.217139 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e29718e2-f954-4ec8-9db2-d6c3638bc51c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4\" (UID: \"e29718e2-f954-4ec8-9db2-d6c3638bc51c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" Nov 03 20:47:20.217214 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.217174 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdvvx\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-kube-api-access-cdvvx\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.217354 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.217203 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:20.217354 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.217283 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-image-registry-private-configuration\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.217354 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.217105 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:47:20.217354 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.217348 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75999699c7-pfc8g: secret "image-registry-tls" not found Nov 03 20:47:20.217552 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.217323 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:20.717302822 +0000 UTC m=+33.318295864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : configmap references non-existent config key: service-ca.crt Nov 03 20:47:20.217552 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.217456 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls podName:793c02dd-468b-4308-bac2-67d04e40579c nodeName:}" failed. No retries permitted until 2026-11-03 20:47:20.717408898 +0000 UTC m=+33.318401923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls") pod "image-registry-75999699c7-pfc8g" (UID: "793c02dd-468b-4308-bac2-67d04e40579c") : secret "image-registry-tls" not found Nov 03 20:47:20.217552 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.217523 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj7td\" (UniqueName: \"kubernetes.io/projected/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-kube-api-access-sj7td\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:20.217683 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.217565 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-certificates\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.217683 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.217592 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-installation-pull-secrets\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.217683 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.217617 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:20.217683 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.217653 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nwhb\" (UniqueName: \"kubernetes.io/projected/ff7262fd-b29d-4c72-b717-ef9742bdd2db-kube-api-access-8nwhb\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:20.217899 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.217693 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mzl9\" (UniqueName: \"kubernetes.io/projected/e29718e2-f954-4ec8-9db2-d6c3638bc51c-kube-api-access-6mzl9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4\" (UID: \"e29718e2-f954-4ec8-9db2-d6c3638bc51c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" Nov 03 20:47:20.217899 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.217719 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-stats-auth\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:20.217899 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.217746 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-trusted-ca\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.217899 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.217747 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/510fbc36-62de-4096-b8e8-d8606a4ad7d5-config\") pod \"service-ca-operator-544d749c8c-s82vc\" (UID: \"510fbc36-62de-4096-b8e8-d8606a4ad7d5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" Nov 03 20:47:20.218115 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.217891 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:20.218115 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.217970 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff7262fd-b29d-4c72-b717-ef9742bdd2db-tmp-dir\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:20.218115 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.218046 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e29718e2-f954-4ec8-9db2-d6c3638bc51c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4\" (UID: \"e29718e2-f954-4ec8-9db2-d6c3638bc51c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" Nov 03 20:47:20.218115 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.218106 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhh48\" (UniqueName: \"kubernetes.io/projected/510fbc36-62de-4096-b8e8-d8606a4ad7d5-kube-api-access-fhh48\") pod \"service-ca-operator-544d749c8c-s82vc\" (UID: \"510fbc36-62de-4096-b8e8-d8606a4ad7d5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" Nov 03 20:47:20.218324 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.218138 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cjnm\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-kube-api-access-5cjnm\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.218324 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.218174 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-default-certificate\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:20.218324 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.218201 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/15d37b2c-d051-419e-b0cb-d761cac6ff74-ca-trust-extracted\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.218324 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.218233 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brl7b\" (UniqueName: \"kubernetes.io/projected/d0d6aa7f-3a2b-482f-8339-291a11a77c22-kube-api-access-brl7b\") pod \"network-check-source-6f78d59f99-t4tng\" (UID: \"d0d6aa7f-3a2b-482f-8339-291a11a77c22\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4tng" Nov 03 20:47:20.218324 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.218270 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nd4m\" (UniqueName: \"kubernetes.io/projected/8f39698b-0204-4492-ac16-e1728e71b8d6-kube-api-access-8nd4m\") pod \"volume-data-source-validator-58c9f5f46d-nm757\" (UID: \"8f39698b-0204-4492-ac16-e1728e71b8d6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nm757" Nov 03 20:47:20.218324 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.218295 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-trusted-ca\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.218324 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.218320 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xwt2z\" (UID: \"fb96b430-8876-41db-a35c-19db3cee7635\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:20.218659 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.218355 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/510fbc36-62de-4096-b8e8-d8606a4ad7d5-serving-cert\") pod \"service-ca-operator-544d749c8c-s82vc\" (UID: \"510fbc36-62de-4096-b8e8-d8606a4ad7d5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" Nov 03 20:47:20.218659 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.218378 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/793c02dd-468b-4308-bac2-67d04e40579c-ca-trust-extracted\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.218659 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.218403 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-installation-pull-secrets\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.218659 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.218407 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/510fbc36-62de-4096-b8e8-d8606a4ad7d5-config\") pod \"service-ca-operator-544d749c8c-s82vc\" (UID: \"510fbc36-62de-4096-b8e8-d8606a4ad7d5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" Nov 03 20:47:20.218659 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.218436 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb96b430-8876-41db-a35c-19db3cee7635-nginx-conf\") pod \"networking-console-plugin-86c478bb88-xwt2z\" (UID: \"fb96b430-8876-41db-a35c-19db3cee7635\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:20.218659 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.218507 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:20.218659 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.218607 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e29718e2-f954-4ec8-9db2-d6c3638bc51c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4\" (UID: \"e29718e2-f954-4ec8-9db2-d6c3638bc51c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" Nov 03 20:47:20.218659 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.218633 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:47:20.219167 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.218757 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:20.718739107 +0000 UTC m=+33.319732174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : secret "router-metrics-certs-default" not found Nov 03 20:47:20.219167 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.218772 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:47:20.219167 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.218819 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls podName:4a6bf982-7a96-41ae-b24d-78cdabbb7c37 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:20.718804365 +0000 UTC m=+33.319797406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-72zbp" (UID: "4a6bf982-7a96-41ae-b24d-78cdabbb7c37") : secret "samples-operator-tls" not found Nov 03 20:47:20.219167 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.218929 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/793c02dd-468b-4308-bac2-67d04e40579c-ca-trust-extracted\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.219167 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.219005 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42d01b00-5ec9-4f8d-a8f2-10e0addd1b67-cert\") pod \"ingress-canary-9c99h\" (UID: \"42d01b00-5ec9-4f8d-a8f2-10e0addd1b67\") " pod="openshift-ingress-canary/ingress-canary-9c99h" Nov 03 20:47:20.219167 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.219142 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-registry-certificates\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.219753 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.219731 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-registry-certificates\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.220487 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.220466 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bd5l5" Nov 03 20:47:20.220943 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.220921 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/510fbc36-62de-4096-b8e8-d8606a4ad7d5-serving-cert\") pod \"service-ca-operator-544d749c8c-s82vc\" (UID: \"510fbc36-62de-4096-b8e8-d8606a4ad7d5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" Nov 03 20:47:20.221584 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.221561 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e29718e2-f954-4ec8-9db2-d6c3638bc51c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4\" (UID: \"e29718e2-f954-4ec8-9db2-d6c3638bc51c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" Nov 03 20:47:20.221824 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.221746 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-default-certificate\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:20.221952 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.221887 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-stats-auth\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:20.223255 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.223233 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-installation-pull-secrets\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.224035 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.224019 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-image-registry-private-configuration\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.272960 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.272756 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:20.283581 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.282138 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj7td\" (UniqueName: \"kubernetes.io/projected/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-kube-api-access-sj7td\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:20.284818 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.284329 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mzl9\" (UniqueName: \"kubernetes.io/projected/e29718e2-f954-4ec8-9db2-d6c3638bc51c-kube-api-access-6mzl9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4\" (UID: \"e29718e2-f954-4ec8-9db2-d6c3638bc51c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" Nov 03 20:47:20.284818 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.284613 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nd4m\" (UniqueName: \"kubernetes.io/projected/8f39698b-0204-4492-ac16-e1728e71b8d6-kube-api-access-8nd4m\") pod \"volume-data-source-validator-58c9f5f46d-nm757\" (UID: \"8f39698b-0204-4492-ac16-e1728e71b8d6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nm757" Nov 03 20:47:20.284818 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.284785 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cjnm\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-kube-api-access-5cjnm\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.285152 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.285095 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-bound-sa-token\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.288158 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.288135 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m4g8\" (UniqueName: \"kubernetes.io/projected/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-kube-api-access-5m4g8\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:20.289139 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.289081 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhh48\" (UniqueName: \"kubernetes.io/projected/510fbc36-62de-4096-b8e8-d8606a4ad7d5-kube-api-access-fhh48\") pod \"service-ca-operator-544d749c8c-s82vc\" (UID: \"510fbc36-62de-4096-b8e8-d8606a4ad7d5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" Nov 03 20:47:20.320929 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.320438 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42d01b00-5ec9-4f8d-a8f2-10e0addd1b67-cert\") pod \"ingress-canary-9c99h\" (UID: \"42d01b00-5ec9-4f8d-a8f2-10e0addd1b67\") " pod="openshift-ingress-canary/ingress-canary-9c99h" Nov 03 20:47:20.320929 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.320473 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7262fd-b29d-4c72-b717-ef9742bdd2db-config-volume\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:20.320929 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.320495 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.320929 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.320516 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fflvq\" (UniqueName: \"kubernetes.io/projected/42d01b00-5ec9-4f8d-a8f2-10e0addd1b67-kube-api-access-fflvq\") pod \"ingress-canary-9c99h\" (UID: \"42d01b00-5ec9-4f8d-a8f2-10e0addd1b67\") " pod="openshift-ingress-canary/ingress-canary-9c99h" Nov 03 20:47:20.320929 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.320542 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-bound-sa-token\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.320929 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.320577 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdvvx\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-kube-api-access-cdvvx\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.320929 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.320615 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-image-registry-private-configuration\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.320929 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.320645 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-certificates\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.320929 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.320665 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-installation-pull-secrets\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.320929 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.320681 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:20.320929 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.320701 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nwhb\" (UniqueName: \"kubernetes.io/projected/ff7262fd-b29d-4c72-b717-ef9742bdd2db-kube-api-access-8nwhb\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:20.320929 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.320730 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff7262fd-b29d-4c72-b717-ef9742bdd2db-tmp-dir\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:20.320929 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.320758 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/15d37b2c-d051-419e-b0cb-d761cac6ff74-ca-trust-extracted\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.320929 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.320774 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brl7b\" (UniqueName: \"kubernetes.io/projected/d0d6aa7f-3a2b-482f-8339-291a11a77c22-kube-api-access-brl7b\") pod \"network-check-source-6f78d59f99-t4tng\" (UID: \"d0d6aa7f-3a2b-482f-8339-291a11a77c22\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4tng" Nov 03 20:47:20.320929 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.320797 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-trusted-ca\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.320929 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.320815 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xwt2z\" (UID: \"fb96b430-8876-41db-a35c-19db3cee7635\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:20.321783 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.320837 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb96b430-8876-41db-a35c-19db3cee7635-nginx-conf\") pod \"networking-console-plugin-86c478bb88-xwt2z\" (UID: \"fb96b430-8876-41db-a35c-19db3cee7635\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:20.321783 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.321083 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:47:20.321783 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.321150 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls podName:ff7262fd-b29d-4c72-b717-ef9742bdd2db nodeName:}" failed. No retries permitted until 2026-11-03 20:47:20.821131158 +0000 UTC m=+33.422124181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls") pod "dns-default-njb7z" (UID: "ff7262fd-b29d-4c72-b717-ef9742bdd2db") : secret "dns-default-metrics-tls" not found Nov 03 20:47:20.321783 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.321279 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff7262fd-b29d-4c72-b717-ef9742bdd2db-tmp-dir\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:20.321783 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.321533 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-certificates\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.322071 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.321936 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:47:20.322071 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.321953 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687954fc7b-25pdn: secret "image-registry-tls" not found Nov 03 20:47:20.322071 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.322006 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls podName:15d37b2c-d051-419e-b0cb-d761cac6ff74 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:20.821988332 +0000 UTC m=+33.422981357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls") pod "image-registry-687954fc7b-25pdn" (UID: "15d37b2c-d051-419e-b0cb-d761cac6ff74") : secret "image-registry-tls" not found Nov 03 20:47:20.322316 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.322292 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7262fd-b29d-4c72-b717-ef9742bdd2db-config-volume\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:20.323001 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.322951 2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:47:20.323239 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.323204 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert podName:fb96b430-8876-41db-a35c-19db3cee7635 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:20.823189403 +0000 UTC m=+33.424182426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xwt2z" (UID: "fb96b430-8876-41db-a35c-19db3cee7635") : secret "networking-console-plugin-cert" not found Nov 03 20:47:20.324240 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.324202 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-trusted-ca\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.324641 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.324583 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/15d37b2c-d051-419e-b0cb-d761cac6ff74-ca-trust-extracted\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.325163 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.325080 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-image-registry-private-configuration\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.325707 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.325662 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42d01b00-5ec9-4f8d-a8f2-10e0addd1b67-cert\") pod \"ingress-canary-9c99h\" (UID: \"42d01b00-5ec9-4f8d-a8f2-10e0addd1b67\") " pod="openshift-ingress-canary/ingress-canary-9c99h" Nov 03 20:47:20.325921 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.325877 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb96b430-8876-41db-a35c-19db3cee7635-nginx-conf\") pod \"networking-console-plugin-86c478bb88-xwt2z\" (UID: \"fb96b430-8876-41db-a35c-19db3cee7635\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:20.326640 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.326621 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-installation-pull-secrets\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.337187 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.337160 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:20.374004 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.373974 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" Nov 03 20:47:20.384272 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.384247 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brl7b\" (UniqueName: \"kubernetes.io/projected/d0d6aa7f-3a2b-482f-8339-291a11a77c22-kube-api-access-brl7b\") pod \"network-check-source-6f78d59f99-t4tng\" (UID: \"d0d6aa7f-3a2b-482f-8339-291a11a77c22\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4tng" Nov 03 20:47:20.384358 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.384247 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fflvq\" (UniqueName: \"kubernetes.io/projected/42d01b00-5ec9-4f8d-a8f2-10e0addd1b67-kube-api-access-fflvq\") pod \"ingress-canary-9c99h\" (UID: \"42d01b00-5ec9-4f8d-a8f2-10e0addd1b67\") " pod="openshift-ingress-canary/ingress-canary-9c99h" Nov 03 20:47:20.384460 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.384440 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-bound-sa-token\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.384769 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.384754 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nwhb\" (UniqueName: \"kubernetes.io/projected/ff7262fd-b29d-4c72-b717-ef9742bdd2db-kube-api-access-8nwhb\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:20.385264 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.385242 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdvvx\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-kube-api-access-cdvvx\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.393789 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.393766 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" Nov 03 20:47:20.397810 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.397785 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nm757" Nov 03 20:47:20.426182 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.426148 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4tng" Nov 03 20:47:20.435113 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.435083 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9c99h" Nov 03 20:47:20.600027 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.599997 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kwzxh"] Nov 03 20:47:20.603021 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:47:20.602984 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea4f6f2_4699_46d7_91bc_8391dde191eb.slice/crio-c5ce2926aaa1b6b38fd605d7bc2906c47d67c146a53ee3e73ff34c8f6862b273 WatchSource:0}: Error finding container c5ce2926aaa1b6b38fd605d7bc2906c47d67c146a53ee3e73ff34c8f6862b273: Status 404 returned error can't find the container with id c5ce2926aaa1b6b38fd605d7bc2906c47d67c146a53ee3e73ff34c8f6862b273 Nov 03 20:47:20.603182 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.603066 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ghxn6"] Nov 03 20:47:20.606130 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:47:20.606107 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ba944d_b672_489d_9acc_fde356202fbe.slice/crio-21441c2dab7222e3b168d973f4aed71f970c88c6ea9028620e10dc238645f40f WatchSource:0}: Error finding container 21441c2dab7222e3b168d973f4aed71f970c88c6ea9028620e10dc238645f40f: Status 404 returned error can't find the container with id 21441c2dab7222e3b168d973f4aed71f970c88c6ea9028620e10dc238645f40f Nov 03 20:47:20.625275 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.624980 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:20.625275 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.625158 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:47:20.625275 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.625251 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls podName:24a97420-efcd-4a6f-b072-1e5792264188 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:21.625227598 +0000 UTC m=+34.226220640 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gkklk" (UID: "24a97420-efcd-4a6f-b072-1e5792264188") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:47:20.700981 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.700884 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-92hts"] Nov 03 20:47:20.720214 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:47:20.720176 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d21a1ae_f712_4f89_b34f_8016fd918484.slice/crio-a7533b85e453b268a4fb61a88978d346221719537f1846cb4d205b1aab1b1451 WatchSource:0}: Error finding container a7533b85e453b268a4fb61a88978d346221719537f1846cb4d205b1aab1b1451: Status 404 returned error can't find the container with id a7533b85e453b268a4fb61a88978d346221719537f1846cb4d205b1aab1b1451 Nov 03 20:47:20.727785 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.726351 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:20.727785 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.726395 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:20.727785 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.726455 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:20.727785 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.726646 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:20.727785 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.726802 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:47:20.727785 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.726931 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls podName:4a6bf982-7a96-41ae-b24d-78cdabbb7c37 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:21.72687102 +0000 UTC m=+34.327864056 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-72zbp" (UID: "4a6bf982-7a96-41ae-b24d-78cdabbb7c37") : secret "samples-operator-tls" not found Nov 03 20:47:20.727785 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.727060 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:21.727042761 +0000 UTC m=+34.328035801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : configmap references non-existent config key: service-ca.crt Nov 03 20:47:20.727785 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.727186 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:47:20.727785 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.727200 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75999699c7-pfc8g: secret "image-registry-tls" not found Nov 03 20:47:20.727785 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.727240 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls podName:793c02dd-468b-4308-bac2-67d04e40579c nodeName:}" failed. No retries permitted until 2026-11-03 20:47:21.727228839 +0000 UTC m=+34.328221876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls") pod "image-registry-75999699c7-pfc8g" (UID: "793c02dd-468b-4308-bac2-67d04e40579c") : secret "image-registry-tls" not found Nov 03 20:47:20.727785 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.727309 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:47:20.727785 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.727345 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:21.727335022 +0000 UTC m=+34.328328060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : secret "router-metrics-certs-default" not found Nov 03 20:47:20.754582 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.754556 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-ccxnp"] Nov 03 20:47:20.756956 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:47:20.756923 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c555a1_e6bd_4042_be58_404cb6693922.slice/crio-c8465dd0f82dfe1e1add0aba8f6155e60ef7df4b52833b40fc8597de1f1faf0f WatchSource:0}: Error finding container c8465dd0f82dfe1e1add0aba8f6155e60ef7df4b52833b40fc8597de1f1faf0f: Status 404 returned error can't find the container with id c8465dd0f82dfe1e1add0aba8f6155e60ef7df4b52833b40fc8597de1f1faf0f Nov 03 20:47:20.771065 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.771038 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bd5l5"] Nov 03 20:47:20.774065 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:47:20.774037 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode863782c_c289_4fa5_93b1_b475358a5001.slice/crio-f79a5c16d5c71a460bfd24e8ccb80dd3a91b4257e178df5f67daa45b0aa5ef5f WatchSource:0}: Error finding container f79a5c16d5c71a460bfd24e8ccb80dd3a91b4257e178df5f67daa45b0aa5ef5f: Status 404 returned error can't find the container with id f79a5c16d5c71a460bfd24e8ccb80dd3a91b4257e178df5f67daa45b0aa5ef5f Nov 03 20:47:20.779725 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.775506 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc"] Nov 03 20:47:20.779725 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:47:20.778377 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510fbc36_62de_4096_b8e8_d8606a4ad7d5.slice/crio-42d007bf2eb83cb4679c66b689dfa99d851c274cfccdd3b74fb8b6be3c440312 WatchSource:0}: Error finding container 42d007bf2eb83cb4679c66b689dfa99d851c274cfccdd3b74fb8b6be3c440312: Status 404 returned error can't find the container with id 42d007bf2eb83cb4679c66b689dfa99d851c274cfccdd3b74fb8b6be3c440312 Nov 03 20:47:20.814063 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.814032 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4"] Nov 03 20:47:20.819222 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.819199 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:20.823626 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:47:20.823593 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29718e2_f954_4ec8_9db2_d6c3638bc51c.slice/crio-8b63b00ad000545bb2d57c0037e6d9f1ac050fbf987df19dd240123684ce4be3 WatchSource:0}: Error finding container 8b63b00ad000545bb2d57c0037e6d9f1ac050fbf987df19dd240123684ce4be3: Status 404 returned error can't find the container with id 8b63b00ad000545bb2d57c0037e6d9f1ac050fbf987df19dd240123684ce4be3 Nov 03 20:47:20.827421 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.827397 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xwt2z\" (UID: \"fb96b430-8876-41db-a35c-19db3cee7635\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:20.827493 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.827465 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:20.827575 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.827563 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:20.827621 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.827589 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:20.828054 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.828035 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:47:20.828122 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.828056 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687954fc7b-25pdn: secret "image-registry-tls" not found Nov 03 20:47:20.828122 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.828103 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:47:20.828211 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.828133 2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:47:20.828211 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.828144 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls podName:15d37b2c-d051-419e-b0cb-d761cac6ff74 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:21.828122353 +0000 UTC m=+34.429115391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls") pod "image-registry-687954fc7b-25pdn" (UID: "15d37b2c-d051-419e-b0cb-d761cac6ff74") : secret "image-registry-tls" not found Nov 03 20:47:20.828211 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.828170 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls podName:ff7262fd-b29d-4c72-b717-ef9742bdd2db nodeName:}" failed. No retries permitted until 2026-11-03 20:47:21.828153131 +0000 UTC m=+34.429146167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls") pod "dns-default-njb7z" (UID: "ff7262fd-b29d-4c72-b717-ef9742bdd2db") : secret "dns-default-metrics-tls" not found Nov 03 20:47:20.828211 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.828198 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert podName:fb96b430-8876-41db-a35c-19db3cee7635 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:21.828189175 +0000 UTC m=+34.429182197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xwt2z" (UID: "fb96b430-8876-41db-a35c-19db3cee7635") : secret "networking-console-plugin-cert" not found Nov 03 20:47:20.830280 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.830260 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:20.837871 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.837815 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nm757"] Nov 03 20:47:20.840498 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:47:20.840470 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f39698b_0204_4492_ac16_e1728e71b8d6.slice/crio-c4dff5b758ed19f952f60182f5f8fe09284a9a1b0e2c433d522537a9e8e38b26 WatchSource:0}: Error finding container c4dff5b758ed19f952f60182f5f8fe09284a9a1b0e2c433d522537a9e8e38b26: Status 404 returned error can't find the container with id c4dff5b758ed19f952f60182f5f8fe09284a9a1b0e2c433d522537a9e8e38b26 Nov 03 20:47:20.869825 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.869798 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-t4tng"] Nov 03 20:47:20.872610 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:47:20.872578 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d6aa7f_3a2b_482f_8339_291a11a77c22.slice/crio-76a98cc0e807c17023de22914cec00e315444bfc62406884a5457bc24da527a5 WatchSource:0}: Error finding container 76a98cc0e807c17023de22914cec00e315444bfc62406884a5457bc24da527a5: Status 404 returned error can't find the container with id 76a98cc0e807c17023de22914cec00e315444bfc62406884a5457bc24da527a5 Nov 03 20:47:20.878376 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.878336 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9c99h"] Nov 03 20:47:20.880340 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:47:20.880319 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d01b00_5ec9_4f8d_a8f2_10e0addd1b67.slice/crio-3f16b832f1a4c5a856e6bf0ca22a73cd23a3e10e3483ed6bd6fe2ff7264d0886 WatchSource:0}: Error finding container 3f16b832f1a4c5a856e6bf0ca22a73cd23a3e10e3483ed6bd6fe2ff7264d0886: Status 404 returned error can't find the container with id 3f16b832f1a4c5a856e6bf0ca22a73cd23a3e10e3483ed6bd6fe2ff7264d0886 Nov 03 20:47:20.907333 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.907306 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:20.928185 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:20.928161 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:20.928328 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.928307 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:47:20.928389 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:20.928372 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs podName:84556453-e5d9-4449-ba59-819c248e1e2a nodeName:}" failed. No retries permitted until 2026-11-03 20:47:52.928358505 +0000 UTC m=+65.529351541 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs") pod "network-metrics-daemon-mr55c" (UID: "84556453-e5d9-4449-ba59-819c248e1e2a") : secret "metrics-daemon-secret" not found Nov 03 20:47:21.029729 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.029695 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd729\" (UniqueName: \"kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729\") pod \"network-check-target-l7lxn\" (UID: \"af51f192-4170-4d3b-85ae-7d2ceb14de05\") " pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:21.032191 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.032167 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd729\" (UniqueName: \"kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729\") pod \"network-check-target-l7lxn\" (UID: \"af51f192-4170-4d3b-85ae-7d2ceb14de05\") " pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:21.043262 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.043229 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:21.111920 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.111878 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bd5l5" event={"ID":"e863782c-c289-4fa5-93b1-b475358a5001","Type":"ContainerStarted","Data":"c3f37b8d49dfef4bece998fe61f6d57d11e38658d436047a7682941c02926651"} Nov 03 20:47:21.112113 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.111960 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bd5l5" event={"ID":"e863782c-c289-4fa5-93b1-b475358a5001","Type":"ContainerStarted","Data":"f79a5c16d5c71a460bfd24e8ccb80dd3a91b4257e178df5f67daa45b0aa5ef5f"} Nov 03 20:47:21.113076 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.113037 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" event={"ID":"e29718e2-f954-4ec8-9db2-d6c3638bc51c","Type":"ContainerStarted","Data":"8b63b00ad000545bb2d57c0037e6d9f1ac050fbf987df19dd240123684ce4be3"} Nov 03 20:47:21.114471 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.114438 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kwzxh" event={"ID":"5ea4f6f2-4699-46d7-91bc-8391dde191eb","Type":"ContainerStarted","Data":"89a63b55c19503105fd7add8413a6d16f9c29c512f9ee4059a4f86639412c206"} Nov 03 20:47:21.114575 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.114482 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kwzxh" event={"ID":"5ea4f6f2-4699-46d7-91bc-8391dde191eb","Type":"ContainerStarted","Data":"c5ce2926aaa1b6b38fd605d7bc2906c47d67c146a53ee3e73ff34c8f6862b273"} Nov 03 20:47:21.115469 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.115449 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4tng" event={"ID":"d0d6aa7f-3a2b-482f-8339-291a11a77c22","Type":"ContainerStarted","Data":"76a98cc0e807c17023de22914cec00e315444bfc62406884a5457bc24da527a5"} Nov 03 20:47:21.116568 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.116548 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" event={"ID":"510fbc36-62de-4096-b8e8-d8606a4ad7d5","Type":"ContainerStarted","Data":"42d007bf2eb83cb4679c66b689dfa99d851c274cfccdd3b74fb8b6be3c440312"} Nov 03 20:47:21.117701 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.117682 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" event={"ID":"e3c555a1-e6bd-4042-be58-404cb6693922","Type":"ContainerStarted","Data":"c8465dd0f82dfe1e1add0aba8f6155e60ef7df4b52833b40fc8597de1f1faf0f"} Nov 03 20:47:21.118816 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.118796 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-92hts" event={"ID":"8d21a1ae-f712-4f89-b34f-8016fd918484","Type":"ContainerStarted","Data":"a7533b85e453b268a4fb61a88978d346221719537f1846cb4d205b1aab1b1451"} Nov 03 20:47:21.120202 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.120183 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ghxn6" event={"ID":"34ba944d-b672-489d-9acc-fde356202fbe","Type":"ContainerStarted","Data":"d46749a4152505e79cd781c7f80ac9218da7579484d848239f2679ea453b3e27"} Nov 03 20:47:21.120282 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.120217 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ghxn6" event={"ID":"34ba944d-b672-489d-9acc-fde356202fbe","Type":"ContainerStarted","Data":"21441c2dab7222e3b168d973f4aed71f970c88c6ea9028620e10dc238645f40f"} Nov 03 20:47:21.121380 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.121360 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nm757" event={"ID":"8f39698b-0204-4492-ac16-e1728e71b8d6","Type":"ContainerStarted","Data":"c4dff5b758ed19f952f60182f5f8fe09284a9a1b0e2c433d522537a9e8e38b26"} Nov 03 20:47:21.122498 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.122468 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9c99h" event={"ID":"42d01b00-5ec9-4f8d-a8f2-10e0addd1b67","Type":"ContainerStarted","Data":"3f16b832f1a4c5a856e6bf0ca22a73cd23a3e10e3483ed6bd6fe2ff7264d0886"} Nov 03 20:47:21.156574 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.156546 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:21.275643 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.275604 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:21.279720 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.279505 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lvbhk"] Nov 03 20:47:21.282229 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:47:21.282187 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1eb3e05_8b22_4992_97d9_c71ecc6b99dc.slice/crio-3e6aa65ba59b277eaf7adf3e04edcd5232a2fee9053f03e8d561441a6c3419cc WatchSource:0}: Error finding container 3e6aa65ba59b277eaf7adf3e04edcd5232a2fee9053f03e8d561441a6c3419cc: Status 404 returned error can't find the container with id 3e6aa65ba59b277eaf7adf3e04edcd5232a2fee9053f03e8d561441a6c3419cc Nov 03 20:47:21.544829 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.544802 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l7lxn"] Nov 03 20:47:21.546917 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:47:21.546876 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf51f192_4170_4d3b_85ae_7d2ceb14de05.slice/crio-0cd0585aea299faded21a33f5383b5092fb1ac9962f5c534580d1feb11a5a58c WatchSource:0}: Error finding container 0cd0585aea299faded21a33f5383b5092fb1ac9962f5c534580d1feb11a5a58c: Status 404 returned error can't find the container with id 0cd0585aea299faded21a33f5383b5092fb1ac9962f5c534580d1feb11a5a58c Nov 03 20:47:21.635830 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.635798 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:21.636017 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:21.635969 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:47:21.636057 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:21.636027 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls podName:24a97420-efcd-4a6f-b072-1e5792264188 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:23.636012977 +0000 UTC m=+36.237006000 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gkklk" (UID: "24a97420-efcd-4a6f-b072-1e5792264188") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:47:21.736753 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.736721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:21.737216 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.736760 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:21.737216 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.736807 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:21.737216 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:21.736955 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:47:21.737216 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.736959 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:21.737216 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:21.736973 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:23.73695347 +0000 UTC m=+36.337946497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : configmap references non-existent config key: service-ca.crt Nov 03 20:47:21.737216 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:21.736984 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:47:21.737216 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:21.736977 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75999699c7-pfc8g: secret "image-registry-tls" not found Nov 03 20:47:21.737216 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:21.737033 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:47:21.737216 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:21.737051 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:23.737033471 +0000 UTC m=+36.338026507 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : secret "router-metrics-certs-default" not found Nov 03 20:47:21.737216 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:21.737068 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls podName:4a6bf982-7a96-41ae-b24d-78cdabbb7c37 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:23.737058795 +0000 UTC m=+36.338051876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-72zbp" (UID: "4a6bf982-7a96-41ae-b24d-78cdabbb7c37") : secret "samples-operator-tls" not found Nov 03 20:47:21.737216 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:21.737080 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls podName:793c02dd-468b-4308-bac2-67d04e40579c nodeName:}" failed. No retries permitted until 2026-11-03 20:47:23.737074296 +0000 UTC m=+36.338067324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls") pod "image-registry-75999699c7-pfc8g" (UID: "793c02dd-468b-4308-bac2-67d04e40579c") : secret "image-registry-tls" not found Nov 03 20:47:21.838335 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.838260 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xwt2z\" (UID: \"fb96b430-8876-41db-a35c-19db3cee7635\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:21.838335 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.838322 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:21.838587 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.838404 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:21.838587 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:21.838423 2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:47:21.838587 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:21.838457 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:47:21.838587 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:21.838471 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687954fc7b-25pdn: secret "image-registry-tls" not found Nov 03 20:47:21.838587 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:21.838503 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert podName:fb96b430-8876-41db-a35c-19db3cee7635 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:23.838483572 +0000 UTC m=+36.439476610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xwt2z" (UID: "fb96b430-8876-41db-a35c-19db3cee7635") : secret "networking-console-plugin-cert" not found Nov 03 20:47:21.838587 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:21.838521 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls podName:15d37b2c-d051-419e-b0cb-d761cac6ff74 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:23.838513913 +0000 UTC m=+36.439506936 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls") pod "image-registry-687954fc7b-25pdn" (UID: "15d37b2c-d051-419e-b0cb-d761cac6ff74") : secret "image-registry-tls" not found Nov 03 20:47:21.838587 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:21.838542 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:47:21.838844 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:21.838634 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls podName:ff7262fd-b29d-4c72-b717-ef9742bdd2db nodeName:}" failed. No retries permitted until 2026-11-03 20:47:23.838617132 +0000 UTC m=+36.439610172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls") pod "dns-default-njb7z" (UID: "ff7262fd-b29d-4c72-b717-ef9742bdd2db") : secret "dns-default-metrics-tls" not found Nov 03 20:47:21.982559 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:21.982494 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ghxn6" podStartSLOduration=412.982475503 podStartE2EDuration="6m52.982475503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:21.98082161 +0000 UTC m=+34.581814653" watchObservedRunningTime="2026-11-03 20:47:21.982475503 +0000 UTC m=+34.583468549" Nov 03 20:47:22.107641 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:22.107532 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-bd5l5" podStartSLOduration=413.107513593 podStartE2EDuration="6m53.107513593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:22.106382364 +0000 UTC m=+34.707375408" watchObservedRunningTime="2026-11-03 20:47:22.107513593 +0000 UTC m=+34.708506638" Nov 03 20:47:22.128994 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:22.128957 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lvbhk" event={"ID":"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc","Type":"ContainerStarted","Data":"3e6aa65ba59b277eaf7adf3e04edcd5232a2fee9053f03e8d561441a6c3419cc"} Nov 03 20:47:22.131593 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:22.131480 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l7lxn" event={"ID":"af51f192-4170-4d3b-85ae-7d2ceb14de05","Type":"ContainerStarted","Data":"0cd0585aea299faded21a33f5383b5092fb1ac9962f5c534580d1feb11a5a58c"} Nov 03 20:47:22.225372 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:22.225312 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-kwzxh" podStartSLOduration=413.225291818 podStartE2EDuration="6m53.225291818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:22.223838386 +0000 UTC m=+34.824831432" watchObservedRunningTime="2026-11-03 20:47:22.225291818 +0000 UTC m=+34.826284860" Nov 03 20:47:22.490521 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:22.490372 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:22.671560 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:22.670693 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:23.656894 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:23.656288 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:23.656894 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:23.656492 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:47:23.656894 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:23.656560 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls podName:24a97420-efcd-4a6f-b072-1e5792264188 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:27.656540914 +0000 UTC m=+40.257533940 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gkklk" (UID: "24a97420-efcd-4a6f-b072-1e5792264188") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:47:23.758578 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:23.757504 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:23.758578 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:23.757589 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:23.758578 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:23.757618 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:23.758578 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:23.757650 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:23.758578 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:23.757824 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:47:23.758578 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:23.757891 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:27.757871707 +0000 UTC m=+40.358864743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : secret "router-metrics-certs-default" not found Nov 03 20:47:23.758578 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:23.758319 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:47:23.758578 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:23.758373 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls podName:4a6bf982-7a96-41ae-b24d-78cdabbb7c37 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:27.758356532 +0000 UTC m=+40.359349568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-72zbp" (UID: "4a6bf982-7a96-41ae-b24d-78cdabbb7c37") : secret "samples-operator-tls" not found Nov 03 20:47:23.758578 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:23.758438 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:27.758427349 +0000 UTC m=+40.359420391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : configmap references non-existent config key: service-ca.crt Nov 03 20:47:23.758578 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:23.758496 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:47:23.758578 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:23.758507 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75999699c7-pfc8g: secret "image-registry-tls" not found Nov 03 20:47:23.758578 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:23.758537 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls podName:793c02dd-468b-4308-bac2-67d04e40579c nodeName:}" failed. No retries permitted until 2026-11-03 20:47:27.758527508 +0000 UTC m=+40.359520543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls") pod "image-registry-75999699c7-pfc8g" (UID: "793c02dd-468b-4308-bac2-67d04e40579c") : secret "image-registry-tls" not found Nov 03 20:47:23.858198 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:23.858157 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xwt2z\" (UID: \"fb96b430-8876-41db-a35c-19db3cee7635\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:23.858387 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:23.858243 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:23.858387 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:23.858350 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:23.858525 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:23.858507 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:47:23.858591 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:23.858579 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls podName:ff7262fd-b29d-4c72-b717-ef9742bdd2db nodeName:}" failed. No retries permitted until 2026-11-03 20:47:27.85856076 +0000 UTC m=+40.459553802 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls") pod "dns-default-njb7z" (UID: "ff7262fd-b29d-4c72-b717-ef9742bdd2db") : secret "dns-default-metrics-tls" not found Nov 03 20:47:23.858666 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:23.858654 2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:47:23.858723 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:23.858694 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert podName:fb96b430-8876-41db-a35c-19db3cee7635 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:27.858682711 +0000 UTC m=+40.459675732 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xwt2z" (UID: "fb96b430-8876-41db-a35c-19db3cee7635") : secret "networking-console-plugin-cert" not found Nov 03 20:47:23.858782 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:23.858763 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:47:23.858782 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:23.858776 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687954fc7b-25pdn: secret "image-registry-tls" not found Nov 03 20:47:23.858879 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:23.858808 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls podName:15d37b2c-d051-419e-b0cb-d761cac6ff74 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:27.858797601 +0000 UTC m=+40.459790641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls") pod "image-registry-687954fc7b-25pdn" (UID: "15d37b2c-d051-419e-b0cb-d761cac6ff74") : secret "image-registry-tls" not found Nov 03 20:47:25.168930 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:25.168111 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nm757" event={"ID":"8f39698b-0204-4492-ac16-e1728e71b8d6","Type":"ContainerStarted","Data":"dca897d6babdf7df2f21133f5b7c1341cc1b91a83782b128249e94e9e5a77b91"} Nov 03 20:47:25.288896 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:25.287334 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nm757" podStartSLOduration=386.161375435 podStartE2EDuration="6m29.28731444s" totalImagesPullingTime="3.125939005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:25.287143307 +0000 UTC m=+37.888136351" watchObservedRunningTime="2026-11-03 20:47:25.28731444 +0000 UTC m=+37.888307485" Nov 03 20:47:25.431612 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:25.431448 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:25.646131 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:25.646058 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:27.179717 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:27.179643 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/0.log" Nov 03 20:47:27.179717 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:27.179695 2583 generic.go:356] "Generic (PLEG): container finished" podID="e3c555a1-e6bd-4042-be58-404cb6693922" containerID="f435eb5e50283cbefd42b451b205640399b7fa98cba08db55d6fe100e2326545" exitCode=255 Nov 03 20:47:27.180138 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:27.179958 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" event={"ID":"e3c555a1-e6bd-4042-be58-404cb6693922","Type":"ContainerDied","Data":"f435eb5e50283cbefd42b451b205640399b7fa98cba08db55d6fe100e2326545"} Nov 03 20:47:27.180138 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:27.180081 2583 scope.go:122] "RemoveContainer" containerID="f435eb5e50283cbefd42b451b205640399b7fa98cba08db55d6fe100e2326545" Nov 03 20:47:27.183118 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:27.183078 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-92hts" event={"ID":"8d21a1ae-f712-4f89-b34f-8016fd918484","Type":"ContainerStarted","Data":"0626089d7672d3fc680a9f4a5533a7d2eeb10627594dde65ce8a9e9e0458e1fd"} Nov 03 20:47:27.424938 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:27.424852 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-92hts" podStartSLOduration=385.302326949 podStartE2EDuration="6m31.424833973s" totalImagesPullingTime="6.122507024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:27.423315477 +0000 UTC m=+40.024308522" watchObservedRunningTime="2026-11-03 20:47:27.424833973 +0000 UTC m=+40.025827021" Nov 03 20:47:27.708525 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:27.708433 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:27.708679 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:27.708621 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:47:27.708731 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:27.708701 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls podName:24a97420-efcd-4a6f-b072-1e5792264188 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:35.708682583 +0000 UTC m=+48.309675612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gkklk" (UID: "24a97420-efcd-4a6f-b072-1e5792264188") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:47:27.809081 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:27.809040 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:27.809081 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:27.809091 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:27.809335 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:27.809125 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:27.809335 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:27.809222 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:35.809198822 +0000 UTC m=+48.410191864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : configmap references non-existent config key: service-ca.crt Nov 03 20:47:27.809335 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:27.809282 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:47:27.809335 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:27.809305 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:27.809335 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:27.809323 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:35.809312285 +0000 UTC m=+48.410305307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : secret "router-metrics-certs-default" not found Nov 03 20:47:27.809567 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:27.809367 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:47:27.809567 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:27.809374 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75999699c7-pfc8g: secret "image-registry-tls" not found Nov 03 20:47:27.809567 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:27.809393 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls podName:793c02dd-468b-4308-bac2-67d04e40579c nodeName:}" failed. No retries permitted until 2026-11-03 20:47:35.809387419 +0000 UTC m=+48.410380441 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls") pod "image-registry-75999699c7-pfc8g" (UID: "793c02dd-468b-4308-bac2-67d04e40579c") : secret "image-registry-tls" not found Nov 03 20:47:27.809567 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:27.809423 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:47:27.809567 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:27.809466 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls podName:4a6bf982-7a96-41ae-b24d-78cdabbb7c37 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:35.809454666 +0000 UTC m=+48.410447690 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-72zbp" (UID: "4a6bf982-7a96-41ae-b24d-78cdabbb7c37") : secret "samples-operator-tls" not found Nov 03 20:47:27.909782 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:27.909747 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:27.909942 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:27.909833 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xwt2z\" (UID: \"fb96b430-8876-41db-a35c-19db3cee7635\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:27.909942 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:27.909918 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:27.910062 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:27.909956 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:47:27.910062 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:27.910024 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls podName:ff7262fd-b29d-4c72-b717-ef9742bdd2db nodeName:}" failed. No retries permitted until 2026-11-03 20:47:35.910005055 +0000 UTC m=+48.510998092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls") pod "dns-default-njb7z" (UID: "ff7262fd-b29d-4c72-b717-ef9742bdd2db") : secret "dns-default-metrics-tls" not found Nov 03 20:47:27.910164 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:27.910094 2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:47:27.910164 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:27.910101 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:47:27.910164 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:27.910116 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687954fc7b-25pdn: secret "image-registry-tls" not found Nov 03 20:47:27.910164 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:27.910130 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert podName:fb96b430-8876-41db-a35c-19db3cee7635 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:35.910118635 +0000 UTC m=+48.511111665 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xwt2z" (UID: "fb96b430-8876-41db-a35c-19db3cee7635") : secret "networking-console-plugin-cert" not found Nov 03 20:47:27.910348 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:27.910177 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls podName:15d37b2c-d051-419e-b0cb-d761cac6ff74 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:35.910160505 +0000 UTC m=+48.511153544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls") pod "image-registry-687954fc7b-25pdn" (UID: "15d37b2c-d051-419e-b0cb-d761cac6ff74") : secret "image-registry-tls" not found Nov 03 20:47:28.055141 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.055100 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d8xh9"] Nov 03 20:47:28.070078 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.070045 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:28.115436 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.115400 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d8xh9"] Nov 03 20:47:28.124121 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.124088 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:47:28.124293 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.124270 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:47:28.124351 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.124324 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mxhpl\"" Nov 03 20:47:28.192699 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.192659 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" event={"ID":"e29718e2-f954-4ec8-9db2-d6c3638bc51c","Type":"ContainerStarted","Data":"473e049556b75a1207b4bf4b4307a644de5e9a451792c92b16e2df7426e2e82c"} Nov 03 20:47:28.194745 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.194648 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" event={"ID":"510fbc36-62de-4096-b8e8-d8606a4ad7d5","Type":"ContainerStarted","Data":"a562f4272a7eea23c0e885af23eb5031cb064fa5ec92af43e2d86743c1020203"} Nov 03 20:47:28.197110 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.197087 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/1.log" Nov 03 20:47:28.197566 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.197551 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/0.log" Nov 03 20:47:28.197638 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.197594 2583 generic.go:356] "Generic (PLEG): container finished" podID="e3c555a1-e6bd-4042-be58-404cb6693922" containerID="29cf45ae230e23b9bed36cf8ecbd1b8c7222aa0b362a7fee1f51cdfc7b569516" exitCode=255 Nov 03 20:47:28.197730 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.197710 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" event={"ID":"e3c555a1-e6bd-4042-be58-404cb6693922","Type":"ContainerDied","Data":"29cf45ae230e23b9bed36cf8ecbd1b8c7222aa0b362a7fee1f51cdfc7b569516"} Nov 03 20:47:28.197783 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.197770 2583 scope.go:122] "RemoveContainer" containerID="f435eb5e50283cbefd42b451b205640399b7fa98cba08db55d6fe100e2326545" Nov 03 20:47:28.197896 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.197879 2583 scope.go:122] "RemoveContainer" containerID="29cf45ae230e23b9bed36cf8ecbd1b8c7222aa0b362a7fee1f51cdfc7b569516" Nov 03 20:47:28.198121 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:28.198096 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ccxnp_openshift-console-operator(e3c555a1-e6bd-4042-be58-404cb6693922)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" podUID="e3c555a1-e6bd-4042-be58-404cb6693922" Nov 03 20:47:28.199569 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.199549 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9c99h" event={"ID":"42d01b00-5ec9-4f8d-a8f2-10e0addd1b67","Type":"ContainerStarted","Data":"0f53fb57a5c4d3ed9af68c9fc7f185afaa1219dfbb5c3a7f805ed8486cda2f15"} Nov 03 20:47:28.221175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.221091 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:28.221175 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.221142 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-data-volume\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:28.221413 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.221233 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:28.221413 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.221303 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l7br\" (UniqueName: \"kubernetes.io/projected/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-api-access-8l7br\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:28.221624 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.221589 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-crio-socket\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:28.290804 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.290770 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-4dvbp-dpsms"] Nov 03 20:47:28.309921 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.309880 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:28.311840 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.311792 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" podStartSLOduration=385.723489203 podStartE2EDuration="6m32.311777614s" totalImagesPullingTime="6.588288411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:28.311205311 +0000 UTC m=+40.912198356" watchObservedRunningTime="2026-11-03 20:47:28.311777614 +0000 UTC m=+40.912770659" Nov 03 20:47:28.317747 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.317342 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:28.322509 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.322469 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-crio-socket\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:28.322660 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.322561 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:28.322660 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.322592 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-data-volume\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:28.322660 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.322640 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:28.322812 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.322743 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l7br\" (UniqueName: \"kubernetes.io/projected/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-api-access-8l7br\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:28.323086 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.323067 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-crio-socket\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:28.324062 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.324038 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-data-volume\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:28.324192 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:28.324062 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:47:28.324192 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:28.324124 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls podName:2955ecad-cad6-48d2-a2f0-1c1aae3d47f4 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:28.82410333 +0000 UTC m=+41.425096352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8xh9" (UID: "2955ecad-cad6-48d2-a2f0-1c1aae3d47f4") : secret "insights-runtime-extractor-tls" not found Nov 03 20:47:28.324335 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:28.324216 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:47:28.324335 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:28.324266 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-rbac-proxy-cm podName:2955ecad-cad6-48d2-a2f0-1c1aae3d47f4 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:28.824247602 +0000 UTC m=+41.425240627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d8xh9" (UID: "2955ecad-cad6-48d2-a2f0-1c1aae3d47f4") : configmap "kube-rbac-proxy" not found Nov 03 20:47:28.349322 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.349283 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4dvbp-dpsms"] Nov 03 20:47:28.387871 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.387840 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l7br\" (UniqueName: \"kubernetes.io/projected/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-api-access-8l7br\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:28.423361 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.423326 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq6p2\" (UniqueName: \"kubernetes.io/projected/1f33677f-ee87-4cd1-bdb0-b74000834db2-kube-api-access-wq6p2\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:28.423531 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.423371 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-trusted-ca-bundle\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:28.423531 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.423460 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1f33677f-ee87-4cd1-bdb0-b74000834db2-archives-path\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:28.423661 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.423641 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-service-ca-bundle\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:28.429742 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.429694 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" podStartSLOduration=385.796630817 podStartE2EDuration="6m32.429681966s" totalImagesPullingTime="6.633051149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:28.428845414 +0000 UTC m=+41.029838459" watchObservedRunningTime="2026-11-03 20:47:28.429681966 +0000 UTC m=+41.030675010" Nov 03 20:47:28.490404 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.490332 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:28.524699 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.524657 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1f33677f-ee87-4cd1-bdb0-b74000834db2-archives-path\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:28.524891 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.524795 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-service-ca-bundle\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:28.524891 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.524845 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq6p2\" (UniqueName: \"kubernetes.io/projected/1f33677f-ee87-4cd1-bdb0-b74000834db2-kube-api-access-wq6p2\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:28.524891 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.524864 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-trusted-ca-bundle\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:28.525181 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.525159 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1f33677f-ee87-4cd1-bdb0-b74000834db2-archives-path\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:28.525356 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.525336 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-service-ca-bundle\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:28.525539 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.525521 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-trusted-ca-bundle\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:28.586798 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.586767 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq6p2\" (UniqueName: \"kubernetes.io/projected/1f33677f-ee87-4cd1-bdb0-b74000834db2-kube-api-access-wq6p2\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:28.648339 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.648305 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:28.668897 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.668351 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9c99h" podStartSLOduration=2.685969117 podStartE2EDuration="9.668331586s" totalImagesPullingTime="6.982362469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:28.665778844 +0000 UTC m=+41.266771888" watchObservedRunningTime="2026-11-03 20:47:28.668331586 +0000 UTC m=+41.269324631" Nov 03 20:47:28.825348 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.825320 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:28.829133 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.829109 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:28.829555 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:28.829165 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:28.829555 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:28.829297 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:47:28.829555 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:28.829351 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:47:28.829555 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:28.829393 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls podName:2955ecad-cad6-48d2-a2f0-1c1aae3d47f4 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:29.829378175 +0000 UTC m=+42.430371208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8xh9" (UID: "2955ecad-cad6-48d2-a2f0-1c1aae3d47f4") : secret "insights-runtime-extractor-tls" not found Nov 03 20:47:28.829555 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:28.829421 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-rbac-proxy-cm podName:2955ecad-cad6-48d2-a2f0-1c1aae3d47f4 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:29.829410526 +0000 UTC m=+42.430403550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d8xh9" (UID: "2955ecad-cad6-48d2-a2f0-1c1aae3d47f4") : configmap "kube-rbac-proxy" not found Nov 03 20:47:29.011441 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.011360 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:29.037804 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.037763 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ndsnt"] Nov 03 20:47:29.040558 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.040538 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ndsnt" Nov 03 20:47:29.047715 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.047687 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qlrbs"] Nov 03 20:47:29.050392 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.050080 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlrbs" Nov 03 20:47:29.064375 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.064351 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4dvbp-dpsms"] Nov 03 20:47:29.069858 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:47:29.069443 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f33677f_ee87_4cd1_bdb0_b74000834db2.slice/crio-c36c6a76d6d9bf65bd809f109b63aed4184fc1227bede75fa9d2ba65a9c1b170 WatchSource:0}: Error finding container c36c6a76d6d9bf65bd809f109b63aed4184fc1227bede75fa9d2ba65a9c1b170: Status 404 returned error can't find the container with id c36c6a76d6d9bf65bd809f109b63aed4184fc1227bede75fa9d2ba65a9c1b170 Nov 03 20:47:29.096467 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.096439 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ndsnt"] Nov 03 20:47:29.098725 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.098702 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 20:47:29.098848 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.098739 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bqd6h\"" Nov 03 20:47:29.098918 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.098869 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 20:47:29.106524 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.106507 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qlrbs"] Nov 03 20:47:29.132924 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.132889 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhvdt\" (UniqueName: \"kubernetes.io/projected/d2af25ca-9ad2-481c-addd-ebe3d999fc4e-kube-api-access-fhvdt\") pod \"migrator-66df5d496b-ndsnt\" (UID: \"d2af25ca-9ad2-481c-addd-ebe3d999fc4e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ndsnt" Nov 03 20:47:29.192959 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.192926 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:29.204821 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.204786 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4dvbp-dpsms" event={"ID":"1f33677f-ee87-4cd1-bdb0-b74000834db2","Type":"ContainerStarted","Data":"82867be803ea01339a9743835e27bce66d65f1aa03767dbabf0164a9be40d687"} Nov 03 20:47:29.204981 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.204843 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4dvbp-dpsms" event={"ID":"1f33677f-ee87-4cd1-bdb0-b74000834db2","Type":"ContainerStarted","Data":"c36c6a76d6d9bf65bd809f109b63aed4184fc1227bede75fa9d2ba65a9c1b170"} Nov 03 20:47:29.206608 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.206581 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4tng" event={"ID":"d0d6aa7f-3a2b-482f-8339-291a11a77c22","Type":"ContainerStarted","Data":"27ce38c07c41370ed65f33d2af30c5593f7ca772067284f302609743433ab6d7"} Nov 03 20:47:29.208548 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.208526 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/1.log" Nov 03 20:47:29.208924 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.208890 2583 scope.go:122] "RemoveContainer" containerID="29cf45ae230e23b9bed36cf8ecbd1b8c7222aa0b362a7fee1f51cdfc7b569516" Nov 03 20:47:29.209134 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:29.209106 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ccxnp_openshift-console-operator(e3c555a1-e6bd-4042-be58-404cb6693922)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" podUID="e3c555a1-e6bd-4042-be58-404cb6693922" Nov 03 20:47:29.212432 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.212406 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l7lxn" event={"ID":"af51f192-4170-4d3b-85ae-7d2ceb14de05","Type":"ContainerStarted","Data":"9a7deddc628d6c6dc323775940b599c9b3278d1de712a0bde1ecc941a914d503"} Nov 03 20:47:29.212816 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.212798 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:29.234065 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.234033 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbvs7\" (UniqueName: \"kubernetes.io/projected/ed5e1db5-564c-4abd-87c4-a2dda264a926-kube-api-access-fbvs7\") pod \"migrator-66df5d496b-qlrbs\" (UID: \"ed5e1db5-564c-4abd-87c4-a2dda264a926\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlrbs" Nov 03 20:47:29.234300 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.234276 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhvdt\" (UniqueName: \"kubernetes.io/projected/d2af25ca-9ad2-481c-addd-ebe3d999fc4e-kube-api-access-fhvdt\") pod \"migrator-66df5d496b-ndsnt\" (UID: \"d2af25ca-9ad2-481c-addd-ebe3d999fc4e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ndsnt" Nov 03 20:47:29.299989 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.299896 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhvdt\" (UniqueName: \"kubernetes.io/projected/d2af25ca-9ad2-481c-addd-ebe3d999fc4e-kube-api-access-fhvdt\") pod \"migrator-66df5d496b-ndsnt\" (UID: \"d2af25ca-9ad2-481c-addd-ebe3d999fc4e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ndsnt" Nov 03 20:47:29.334969 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.334934 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbvs7\" (UniqueName: \"kubernetes.io/projected/ed5e1db5-564c-4abd-87c4-a2dda264a926-kube-api-access-fbvs7\") pod \"migrator-66df5d496b-qlrbs\" (UID: \"ed5e1db5-564c-4abd-87c4-a2dda264a926\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlrbs" Nov 03 20:47:29.354664 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.354636 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ndsnt" Nov 03 20:47:29.378847 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.378766 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:29.404356 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.404323 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbvs7\" (UniqueName: \"kubernetes.io/projected/ed5e1db5-564c-4abd-87c4-a2dda264a926-kube-api-access-fbvs7\") pod \"migrator-66df5d496b-qlrbs\" (UID: \"ed5e1db5-564c-4abd-87c4-a2dda264a926\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlrbs" Nov 03 20:47:29.515656 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.515584 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-l7lxn" podStartSLOduration=33.239472679 podStartE2EDuration="40.515565226s" totalImagesPullingTime="7.276092547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:29.515232805 +0000 UTC m=+42.116225850" watchObservedRunningTime="2026-11-03 20:47:29.515565226 +0000 UTC m=+42.116558271" Nov 03 20:47:29.632835 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.632724 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4tng" podStartSLOduration=382.824845136 podStartE2EDuration="6m30.632704079s" totalImagesPullingTime="7.807858943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:29.632086191 +0000 UTC m=+42.233079237" watchObservedRunningTime="2026-11-03 20:47:29.632704079 +0000 UTC m=+42.233697127" Nov 03 20:47:29.664730 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.664691 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlrbs" Nov 03 20:47:29.733966 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.733937 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ndsnt"] Nov 03 20:47:29.744570 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:47:29.744446 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2af25ca_9ad2_481c_addd_ebe3d999fc4e.slice/crio-03ec5291801f3033463914cf526c6a4245674a2fa6b82dc5e8f69dfba3a9d552 WatchSource:0}: Error finding container 03ec5291801f3033463914cf526c6a4245674a2fa6b82dc5e8f69dfba3a9d552: Status 404 returned error can't find the container with id 03ec5291801f3033463914cf526c6a4245674a2fa6b82dc5e8f69dfba3a9d552 Nov 03 20:47:29.751961 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.750931 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-4dvbp-dpsms" podStartSLOduration=1.750857997 podStartE2EDuration="1.750857997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:29.74969345 +0000 UTC m=+42.350686496" watchObservedRunningTime="2026-11-03 20:47:29.750857997 +0000 UTC m=+42.351851033" Nov 03 20:47:29.841373 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.841338 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:29.841573 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:29.841395 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:29.841573 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:29.841499 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:47:29.841573 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:29.841517 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:47:29.841573 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:29.841566 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-rbac-proxy-cm podName:2955ecad-cad6-48d2-a2f0-1c1aae3d47f4 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:31.841547497 +0000 UTC m=+44.442540533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d8xh9" (UID: "2955ecad-cad6-48d2-a2f0-1c1aae3d47f4") : configmap "kube-rbac-proxy" not found Nov 03 20:47:29.841788 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:29.841586 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls podName:2955ecad-cad6-48d2-a2f0-1c1aae3d47f4 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:31.841575351 +0000 UTC m=+44.442568377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8xh9" (UID: "2955ecad-cad6-48d2-a2f0-1c1aae3d47f4") : secret "insights-runtime-extractor-tls" not found Nov 03 20:47:30.046350 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.045834 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qlrbs"] Nov 03 20:47:30.048051 ip-10-0-130-134 kubenswrapper[2583]: W1103 20:47:30.048019 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5e1db5_564c_4abd_87c4_a2dda264a926.slice/crio-070afec86d4588d7b7081c46e8bea71da9940391ce8b8a1f1f332305085ebca7 WatchSource:0}: Error finding container 070afec86d4588d7b7081c46e8bea71da9940391ce8b8a1f1f332305085ebca7: Status 404 returned error can't find the container with id 070afec86d4588d7b7081c46e8bea71da9940391ce8b8a1f1f332305085ebca7 Nov 03 20:47:30.190160 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.190124 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-vcjdr"] Nov 03 20:47:30.193428 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.193407 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" Nov 03 20:47:30.217852 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.217782 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlrbs" event={"ID":"ed5e1db5-564c-4abd-87c4-a2dda264a926","Type":"ContainerStarted","Data":"070afec86d4588d7b7081c46e8bea71da9940391ce8b8a1f1f332305085ebca7"} Nov 03 20:47:30.219261 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.219233 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ndsnt" event={"ID":"d2af25ca-9ad2-481c-addd-ebe3d999fc4e","Type":"ContainerStarted","Data":"03ec5291801f3033463914cf526c6a4245674a2fa6b82dc5e8f69dfba3a9d552"} Nov 03 20:47:30.248041 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.247610 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-54hgz\"" Nov 03 20:47:30.248041 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.247729 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 20:47:30.248041 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.247806 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 20:47:30.248041 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.247829 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 20:47:30.248041 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.247874 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 20:47:30.248343 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.248087 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-vcjdr"] Nov 03 20:47:30.338508 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.338421 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:30.338508 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.338488 2583 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:30.338992 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.338971 2583 scope.go:122] "RemoveContainer" containerID="29cf45ae230e23b9bed36cf8ecbd1b8c7222aa0b362a7fee1f51cdfc7b569516" Nov 03 20:47:30.339231 ip-10-0-130-134 kubenswrapper[2583]: E1103 20:47:30.339207 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ccxnp_openshift-console-operator(e3c555a1-e6bd-4042-be58-404cb6693922)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" podUID="e3c555a1-e6bd-4042-be58-404cb6693922" Nov 03 20:47:30.346500 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.346465 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bxrc\" (UniqueName: \"kubernetes.io/projected/12ce6198-3748-49b2-ad3d-f5c3d701fc96-kube-api-access-6bxrc\") pod \"service-ca-5fffc97bdf-vcjdr\" (UID: \"12ce6198-3748-49b2-ad3d-f5c3d701fc96\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" Nov 03 20:47:30.346678 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.346655 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/12ce6198-3748-49b2-ad3d-f5c3d701fc96-signing-key\") pod \"service-ca-5fffc97bdf-vcjdr\" (UID: \"12ce6198-3748-49b2-ad3d-f5c3d701fc96\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" Nov 03 20:47:30.346799 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.346741 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/12ce6198-3748-49b2-ad3d-f5c3d701fc96-signing-cabundle\") pod \"service-ca-5fffc97bdf-vcjdr\" (UID: \"12ce6198-3748-49b2-ad3d-f5c3d701fc96\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" Nov 03 20:47:30.448126 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.448078 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bxrc\" (UniqueName: \"kubernetes.io/projected/12ce6198-3748-49b2-ad3d-f5c3d701fc96-kube-api-access-6bxrc\") pod \"service-ca-5fffc97bdf-vcjdr\" (UID: \"12ce6198-3748-49b2-ad3d-f5c3d701fc96\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" Nov 03 20:47:30.448296 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.448202 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/12ce6198-3748-49b2-ad3d-f5c3d701fc96-signing-key\") pod \"service-ca-5fffc97bdf-vcjdr\" (UID: \"12ce6198-3748-49b2-ad3d-f5c3d701fc96\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" Nov 03 20:47:30.448296 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.448237 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/12ce6198-3748-49b2-ad3d-f5c3d701fc96-signing-cabundle\") pod \"service-ca-5fffc97bdf-vcjdr\" (UID: \"12ce6198-3748-49b2-ad3d-f5c3d701fc96\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" Nov 03 20:47:30.449057 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.449015 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/12ce6198-3748-49b2-ad3d-f5c3d701fc96-signing-cabundle\") pod \"service-ca-5fffc97bdf-vcjdr\" (UID: \"12ce6198-3748-49b2-ad3d-f5c3d701fc96\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" Nov 03 20:47:30.451320 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.451293 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/12ce6198-3748-49b2-ad3d-f5c3d701fc96-signing-key\") pod \"service-ca-5fffc97bdf-vcjdr\" (UID: \"12ce6198-3748-49b2-ad3d-f5c3d701fc96\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" Nov 03 20:47:30.479412 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.479379 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:30.510751 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.510693 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bxrc\" (UniqueName: \"kubernetes.io/projected/12ce6198-3748-49b2-ad3d-f5c3d701fc96-kube-api-access-6bxrc\") pod \"service-ca-5fffc97bdf-vcjdr\" (UID: \"12ce6198-3748-49b2-ad3d-f5c3d701fc96\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" Nov 03 20:47:30.692595 ip-10-0-130-134 kubenswrapper[2583]: I1103 20:47:30.692565 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:30.753102 ip-10-0-130-134 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 20:47:30.772420 ip-10-0-130-134 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 20:47:30.772621 ip-10-0-130-134 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 20:47:30.772682 ip-10-0-130-134 systemd[1]: kubelet.service: Consumed 3.310s CPU time. Nov 03 20:47:30.790636 ip-10-0-130-134 systemd[1]: Starting Kubernetes Kubelet... Nov 03 20:47:30.870557 ip-10-0-130-134 kubenswrapper[4963]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:47:30.870557 ip-10-0-130-134 kubenswrapper[4963]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 20:47:30.870557 ip-10-0-130-134 kubenswrapper[4963]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:47:30.870557 ip-10-0-130-134 kubenswrapper[4963]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:47:30.872935 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872807 4963 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:47:30.872935 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872934 4963 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:47:30.872935 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872938 4963 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:47:30.872935 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872942 4963 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872946 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872949 4963 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872952 4963 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872955 4963 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872963 4963 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872966 4963 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872969 4963 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872973 4963 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872975 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872978 4963 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872980 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872983 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872986 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872988 4963 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872991 4963 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872993 4963 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872996 4963 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.872999 4963 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873002 4963 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:47:30.873061 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873004 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873009 4963 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873013 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873017 4963 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873020 4963 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873022 4963 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873025 4963 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873028 4963 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873031 4963 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873035 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873038 4963 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873041 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873044 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873046 4963 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873049 4963 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873052 4963 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873056 4963 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873058 4963 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873061 4963 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:47:30.873550 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873064 4963 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873066 4963 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873069 4963 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873072 4963 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873074 4963 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873077 4963 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873080 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873083 4963 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873085 4963 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873088 4963 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873090 4963 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873093 4963 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873096 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873099 4963 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873102 4963 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873105 4963 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873108 4963 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873111 4963 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873113 4963 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873116 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:47:30.874035 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873119 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873123 4963 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873126 4963 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873128 4963 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873131 4963 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873134 4963 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873137 4963 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873139 4963 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873142 4963 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873145 4963 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873148 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873150 4963 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873153 4963 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873155 4963 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873158 4963 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873161 4963 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873163 4963 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873167 4963 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873169 4963 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873172 4963 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:47:30.874508 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873175 4963 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873178 4963 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873180 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873183 4963 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873186 4963 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873189 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873195 4963 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873198 4963 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873201 4963 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873205 4963 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873207 4963 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873210 4963 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873212 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873215 4963 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873218 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873220 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873223 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873226 4963 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873228 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:47:30.875006 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873231 4963 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873234 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873237 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873240 4963 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.873244 4963 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874118 4963 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874126 4963 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874135 4963 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874139 4963 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874142 4963 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874144 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874147 4963 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874150 4963 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874152 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874155 4963 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874158 4963 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874161 4963 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874164 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874167 4963 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:47:30.875469 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874170 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874172 4963 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874175 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874178 4963 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874180 4963 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874183 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874185 4963 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874188 4963 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874190 4963 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874193 4963 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874196 4963 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874198 4963 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874201 4963 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874204 4963 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874206 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874209 4963 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874212 4963 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874214 4963 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874222 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:47:30.875946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874225 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874228 4963 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874230 4963 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874233 4963 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874236 4963 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874238 4963 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874241 4963 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874244 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874247 4963 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874250 4963 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874253 4963 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874255 4963 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874259 4963 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874262 4963 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874264 4963 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874267 4963 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874269 4963 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874272 4963 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874275 4963 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874277 4963 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:47:30.876433 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874280 4963 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874282 4963 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874285 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874288 4963 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874290 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874293 4963 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874295 4963 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874298 4963 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874300 4963 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874303 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874305 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874309 4963 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874318 4963 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874324 4963 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874327 4963 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874330 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874333 4963 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874336 4963 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874339 4963 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874342 4963 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:47:30.876929 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874344 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874347 4963 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874350 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874353 4963 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874355 4963 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874358 4963 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874360 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874363 4963 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874365 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874368 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874371 4963 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874373 4963 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874376 4963 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874380 4963 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874383 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874386 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874388 4963 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874391 4963 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874394 4963 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:47:30.877453 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874396 4963 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874399 4963 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874401 4963 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874404 4963 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874407 4963 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874409 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874413 4963 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874416 4963 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874418 4963 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874421 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874424 4963 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874427 4963 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874430 4963 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.874432 4963 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875005 4963 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875011 4963 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875015 4963 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875018 4963 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875020 4963 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875023 4963 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:47:30.877952 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875026 4963 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875028 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875031 4963 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875034 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875037 4963 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875039 4963 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875042 4963 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875045 4963 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875048 4963 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875051 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875053 4963 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875056 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875059 4963 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875061 4963 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875064 4963 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875066 4963 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875069 4963 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875071 4963 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875080 4963 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:47:30.878445 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875083 4963 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875086 4963 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875089 4963 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875091 4963 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875094 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875096 4963 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875099 4963 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875106 4963 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875109 4963 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875111 4963 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875114 4963 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875116 4963 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875119 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875121 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875124 4963 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875128 4963 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875130 4963 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875133 4963 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875136 4963 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875138 4963 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:47:30.878973 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875141 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875143 4963 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875146 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875148 4963 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875151 4963 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875154 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875156 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875159 4963 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875161 4963 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875164 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875167 4963 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875169 4963 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875174 4963 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875178 4963 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875182 4963 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875185 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875189 4963 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875193 4963 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875196 4963 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:47:30.879812 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875201 4963 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875204 4963 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875207 4963 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875209 4963 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875212 4963 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875215 4963 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875217 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875220 4963 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875225 4963 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875228 4963 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875231 4963 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875233 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875236 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875239 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875241 4963 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875244 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875247 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875250 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875252 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:47:30.880858 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875255 4963 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875257 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875260 4963 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875263 4963 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875265 4963 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875268 4963 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875270 4963 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875274 4963 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875276 4963 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875279 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875281 4963 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875284 4963 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875288 4963 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875292 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875294 4963 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875297 4963 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875300 4963 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875302 4963 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875305 4963 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875307 4963 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:47:30.881373 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875310 4963 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875314 4963 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.875316 4963 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875423 4963 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875432 4963 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875443 4963 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875449 4963 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875453 4963 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875456 4963 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875461 4963 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875466 4963 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875469 4963 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875472 4963 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875476 4963 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875480 4963 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875483 4963 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875486 4963 flags.go:64] FLAG: --cgroup-root="" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875489 4963 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875493 4963 flags.go:64] FLAG: --client-ca-file="" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875496 4963 flags.go:64] FLAG: --cloud-provider="external" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875500 4963 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875504 4963 flags.go:64] FLAG: --cluster-domain="" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875507 4963 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875510 4963 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 20:47:30.881871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875513 4963 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875518 4963 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875522 4963 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875525 4963 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875528 4963 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875532 4963 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875535 4963 flags.go:64] FLAG: --contention-profiling="false" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875538 4963 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875541 4963 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875546 4963 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875549 4963 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875554 4963 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875557 4963 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875560 4963 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875563 4963 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875566 4963 flags.go:64] FLAG: --enable-server="true" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875569 4963 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875574 4963 flags.go:64] FLAG: --event-burst="100" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875578 4963 flags.go:64] FLAG: --event-qps="50" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875581 4963 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875584 4963 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875587 4963 flags.go:64] FLAG: --eviction-hard="" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875592 4963 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875595 4963 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875598 4963 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 20:47:30.882499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875601 4963 flags.go:64] FLAG: --eviction-soft="" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875605 4963 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875608 4963 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875611 4963 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875614 4963 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875618 4963 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875621 4963 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875624 4963 flags.go:64] FLAG: --feature-gates="" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875629 4963 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875632 4963 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875635 4963 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875638 4963 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875642 4963 flags.go:64] FLAG: --healthz-port="10248" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875645 4963 flags.go:64] FLAG: --help="false" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875648 4963 flags.go:64] FLAG: --hostname-override="ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875652 4963 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875656 4963 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875659 4963 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875663 4963 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875666 4963 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875669 4963 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875672 4963 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875675 4963 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 20:47:30.883125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875678 4963 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875682 4963 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875685 4963 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875688 4963 flags.go:64] FLAG: --kube-reserved="" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875691 4963 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875694 4963 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875697 4963 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875700 4963 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875704 4963 flags.go:64] FLAG: --lock-file="" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875707 4963 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875710 4963 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875713 4963 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875718 4963 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875721 4963 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875725 4963 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875728 4963 flags.go:64] FLAG: --logging-format="text" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875731 4963 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875735 4963 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875738 4963 flags.go:64] FLAG: --manifest-url="" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875741 4963 flags.go:64] FLAG: --manifest-url-header="" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875746 4963 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875749 4963 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875753 4963 flags.go:64] FLAG: --max-pods="110" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875756 4963 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875759 4963 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 20:47:30.883684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875763 4963 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875767 4963 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875770 4963 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875773 4963 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875776 4963 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875783 4963 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875787 4963 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875790 4963 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875793 4963 flags.go:64] FLAG: --pod-cidr="" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875796 4963 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875799 4963 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875802 4963 flags.go:64] FLAG: --pods-per-core="0" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875805 4963 flags.go:64] FLAG: --port="10250" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875808 4963 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875811 4963 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0f708d82c0704dc34" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875815 4963 flags.go:64] FLAG: --qos-reserved="" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875818 4963 flags.go:64] FLAG: --read-only-port="10255" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875821 4963 flags.go:64] FLAG: --register-node="true" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875824 4963 flags.go:64] FLAG: --register-with-taints="" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875828 4963 flags.go:64] FLAG: --registry-burst="10" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875831 4963 flags.go:64] FLAG: --registry-qps="5" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875835 4963 flags.go:64] FLAG: --reserved-cpus="" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875838 4963 flags.go:64] FLAG: --reserved-memory="" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875842 4963 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875845 4963 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875849 4963 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 20:47:30.884294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875852 4963 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875855 4963 flags.go:64] FLAG: --runonce="false" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875858 4963 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875861 4963 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875864 4963 flags.go:64] FLAG: --seccomp-default="false" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875866 4963 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875872 4963 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875875 4963 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875878 4963 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875882 4963 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875885 4963 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875888 4963 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875891 4963 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875894 4963 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875913 4963 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875919 4963 flags.go:64] FLAG: --system-cgroups="" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875922 4963 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875928 4963 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875931 4963 flags.go:64] FLAG: --tls-cert-file="" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875934 4963 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875938 4963 flags.go:64] FLAG: --tls-min-version="" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875941 4963 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875944 4963 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875947 4963 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875950 4963 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 20:47:30.884935 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875954 4963 flags.go:64] FLAG: --v="2" Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875958 4963 flags.go:64] FLAG: --version="false" Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875964 4963 flags.go:64] FLAG: --vmodule="" Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875969 4963 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.875973 4963 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876113 4963 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876117 4963 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876120 4963 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876122 4963 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876125 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876128 4963 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876131 4963 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876133 4963 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876138 4963 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876140 4963 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876143 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876146 4963 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876148 4963 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876151 4963 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876153 4963 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:47:30.885574 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876156 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876160 4963 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876163 4963 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876165 4963 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876168 4963 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876170 4963 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876173 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876177 4963 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876181 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876184 4963 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876187 4963 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876190 4963 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876192 4963 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876195 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876198 4963 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876201 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876203 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876206 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876209 4963 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:47:30.886080 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876212 4963 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876214 4963 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876217 4963 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876219 4963 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876222 4963 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876224 4963 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876229 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876231 4963 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876234 4963 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876236 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876239 4963 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876244 4963 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876247 4963 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876252 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876256 4963 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876259 4963 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876262 4963 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876265 4963 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876267 4963 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:47:30.886540 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876270 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876273 4963 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876276 4963 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876278 4963 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876281 4963 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876284 4963 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876287 4963 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876290 4963 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876292 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876295 4963 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876297 4963 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876300 4963 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876303 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876306 4963 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876308 4963 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876311 4963 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876313 4963 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876316 4963 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876319 4963 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:47:30.887162 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876323 4963 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876326 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876329 4963 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876331 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876334 4963 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876337 4963 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876339 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876342 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876346 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876352 4963 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876355 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876357 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876360 4963 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876362 4963 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876365 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876368 4963 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876370 4963 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876373 4963 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876376 4963 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:47:30.887720 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876378 4963 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:47:30.888462 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876381 4963 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:47:30.888462 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876383 4963 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:47:30.888462 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876386 4963 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:47:30.888462 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876389 4963 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:47:30.888462 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876391 4963 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:47:30.888462 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876394 4963 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:47:30.888462 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876397 4963 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:47:30.888462 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876400 4963 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:47:30.888462 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876402 4963 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:47:30.888462 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876405 4963 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:47:30.888462 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876407 4963 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:47:30.888462 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876410 4963 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:47:30.888462 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876415 4963 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:47:30.888462 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876417 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:47:30.888462 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.876447 4963 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:47:30.888462 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876484 4963 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:47:30.888462 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876489 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876492 4963 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876495 4963 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876499 4963 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876503 4963 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876506 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876508 4963 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876511 4963 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876514 4963 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876516 4963 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876519 4963 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876522 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876524 4963 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876527 4963 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876530 4963 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876532 4963 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876535 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876537 4963 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876540 4963 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876543 4963 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:47:30.889185 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876545 4963 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876548 4963 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876550 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876555 4963 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876559 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876562 4963 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876564 4963 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876567 4963 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876569 4963 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876574 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876577 4963 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876579 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876582 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876585 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876587 4963 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876590 4963 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876594 4963 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876596 4963 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876599 4963 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:47:30.889741 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876601 4963 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876604 4963 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876607 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876610 4963 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876612 4963 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876615 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876617 4963 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876621 4963 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876624 4963 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876627 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876630 4963 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876633 4963 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876636 4963 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876639 4963 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876642 4963 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876644 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876647 4963 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876650 4963 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876652 4963 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:47:30.890304 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876656 4963 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876659 4963 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876662 4963 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876666 4963 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876669 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876672 4963 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876674 4963 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876677 4963 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876680 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876683 4963 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876687 4963 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876689 4963 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876692 4963 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876694 4963 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876697 4963 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876699 4963 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876702 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876704 4963 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876707 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:47:30.890997 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876710 4963 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876712 4963 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876714 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876717 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876720 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876722 4963 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876725 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876727 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876730 4963 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876733 4963 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876735 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876738 4963 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876741 4963 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876743 4963 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876746 4963 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876749 4963 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876752 4963 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876755 4963 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876758 4963 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876761 4963 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:47:30.891617 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876764 4963 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876767 4963 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876769 4963 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876776 4963 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876779 4963 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876781 4963 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876784 4963 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.876787 4963 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.882478 4963 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.882500 4963 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.882525 4963 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.882531 4963 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882598 4963 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882604 4963 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882608 4963 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882611 4963 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882614 4963 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882618 4963 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882621 4963 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:47:30.892159 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882623 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882628 4963 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882632 4963 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882636 4963 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882639 4963 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882642 4963 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882646 4963 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882651 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882655 4963 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882658 4963 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882661 4963 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882664 4963 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882667 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882669 4963 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882672 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882674 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882677 4963 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882680 4963 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882683 4963 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:47:30.892620 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882685 4963 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882688 4963 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882690 4963 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882693 4963 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882696 4963 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882698 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882701 4963 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882704 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882706 4963 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882709 4963 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882712 4963 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882715 4963 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882717 4963 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882720 4963 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882723 4963 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882725 4963 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882728 4963 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882730 4963 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882733 4963 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882736 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:47:30.893093 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882744 4963 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882747 4963 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882750 4963 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882753 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882755 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882758 4963 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882760 4963 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882763 4963 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882766 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882770 4963 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882772 4963 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882775 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882778 4963 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882780 4963 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882783 4963 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882786 4963 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882788 4963 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882791 4963 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882793 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882796 4963 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:47:30.893583 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882799 4963 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882801 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882804 4963 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882806 4963 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882809 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882811 4963 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882814 4963 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882816 4963 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882819 4963 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882821 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882824 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882827 4963 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882830 4963 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882834 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882836 4963 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882839 4963 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882842 4963 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882845 4963 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882847 4963 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:47:30.894451 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882850 4963 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882852 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882855 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882858 4963 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882861 4963 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882863 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882866 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882868 4963 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882871 4963 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882874 4963 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882876 4963 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882879 4963 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882881 4963 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882884 4963 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882887 4963 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882890 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882892 4963 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882895 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882897 4963 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882916 4963 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:47:30.895145 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.882919 4963 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:47:30.895736 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.882952 4963 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:47:30.895736 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883098 4963 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:47:30.895736 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883102 4963 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:47:30.895736 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883106 4963 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:47:30.895736 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883109 4963 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:47:30.895736 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883112 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:47:30.895736 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883116 4963 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:47:30.895736 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883119 4963 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:47:30.895736 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883121 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:47:30.895736 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883124 4963 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:47:30.895736 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883127 4963 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:47:30.895736 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883130 4963 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:47:30.895736 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883132 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:47:30.895736 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883135 4963 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:47:30.895736 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883138 4963 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883143 4963 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883147 4963 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883151 4963 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883154 4963 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883157 4963 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883160 4963 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883163 4963 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883166 4963 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883168 4963 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883171 4963 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883174 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883176 4963 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883179 4963 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883182 4963 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883184 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883187 4963 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883190 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883192 4963 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:47:30.896211 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883195 4963 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883197 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883200 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883202 4963 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883206 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883209 4963 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883212 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883215 4963 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883218 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883220 4963 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883223 4963 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883226 4963 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883229 4963 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883231 4963 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883234 4963 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883236 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883239 4963 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883242 4963 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883244 4963 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:47:30.896815 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883247 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883249 4963 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883252 4963 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883254 4963 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883257 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883259 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883262 4963 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883265 4963 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883267 4963 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883270 4963 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883273 4963 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883275 4963 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883278 4963 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883281 4963 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883283 4963 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883286 4963 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883288 4963 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883292 4963 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883295 4963 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883297 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:47:30.897577 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883300 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883303 4963 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883306 4963 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883309 4963 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883311 4963 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883314 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883316 4963 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883319 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883321 4963 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883324 4963 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883326 4963 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883329 4963 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883331 4963 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883334 4963 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883336 4963 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883340 4963 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883343 4963 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883347 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883350 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:47:30.898289 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883352 4963 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:47:30.898946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883355 4963 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:47:30.898946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883358 4963 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:47:30.898946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883360 4963 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:47:30.898946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883363 4963 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:47:30.898946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883365 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:47:30.898946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883368 4963 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:47:30.898946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883371 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:47:30.898946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883373 4963 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:47:30.898946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883376 4963 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:47:30.898946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883380 4963 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:47:30.898946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883382 4963 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:47:30.898946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883385 4963 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:47:30.898946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883388 4963 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:47:30.898946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883391 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:47:30.898946 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883394 4963 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:47:30.898946 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.883416 4963 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883449 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883453 4963 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883456 4963 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883459 4963 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883462 4963 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883465 4963 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883467 4963 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883470 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883473 4963 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883475 4963 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883478 4963 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883481 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883484 4963 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883486 4963 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883489 4963 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883492 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883494 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883497 4963 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883500 4963 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:47:30.899447 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883502 4963 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883505 4963 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883507 4963 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883510 4963 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883513 4963 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883515 4963 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883518 4963 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883521 4963 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883524 4963 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883527 4963 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883529 4963 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883532 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883535 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883537 4963 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883541 4963 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883544 4963 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883546 4963 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883549 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883551 4963 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883554 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:47:30.900055 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883556 4963 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883559 4963 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883561 4963 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883564 4963 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883566 4963 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883569 4963 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883571 4963 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883575 4963 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883578 4963 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883581 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883584 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883586 4963 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883589 4963 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883591 4963 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883594 4963 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883597 4963 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883599 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883602 4963 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883605 4963 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:47:30.900846 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883607 4963 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883610 4963 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883612 4963 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883615 4963 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883618 4963 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883621 4963 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883623 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883626 4963 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883628 4963 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883631 4963 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883633 4963 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883636 4963 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883639 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883641 4963 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883644 4963 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883646 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883649 4963 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883651 4963 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883654 4963 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:47:30.901391 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883656 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883659 4963 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883661 4963 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883665 4963 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883669 4963 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883673 4963 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883675 4963 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883678 4963 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883685 4963 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883688 4963 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883691 4963 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883694 4963 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883697 4963 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883700 4963 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883703 4963 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883705 4963 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883708 4963 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883712 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883715 4963 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883718 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:47:30.902025 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883721 4963 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:47:30.902522 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883723 4963 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:47:30.902522 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883726 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:47:30.902522 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883729 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:47:30.902522 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883731 4963 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:47:30.902522 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883734 4963 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:47:30.902522 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883736 4963 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:47:30.902522 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883739 4963 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:47:30.902522 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:30.883741 4963 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:47:30.902522 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.883807 4963 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 20:47:30.902522 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.884867 4963 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 20:47:30.902522 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.884949 4963 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 20:47:30.902522 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.885138 4963 server.go:1014] "Starting client certificate rotation" Nov 03 20:47:30.902522 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.885274 4963 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:47:30.902522 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.885317 4963 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:41:48 +0000 UTC" deadline="2028-04-09 10:53:40.460630151 +0000 UTC" Nov 03 20:47:30.902522 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.885346 4963 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12542h6m9.575285743s" Nov 03 20:47:30.902522 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.885602 4963 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:47:30.902933 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.886745 4963 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:47:30.902933 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.888141 4963 log.go:25] "Validated CRI v1 runtime API" Nov 03 20:47:30.902933 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.890096 4963 log.go:25] "Validated CRI v1 image API" Nov 03 20:47:30.902933 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.890815 4963 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 20:47:30.902933 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.893333 4963 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c79ab3fe-ba33-4c36-b0d9-d0963013f182:/dev/nvme0n1p4 e463d441-011b-4dd7-9143-60ee9243974a:/dev/nvme0n1p3] Nov 03 20:47:30.903102 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.893353 4963 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03ec5291801f3033463914cf526c6a4245674a2fa6b82dc5e8f69dfba3a9d552/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03ec5291801f3033463914cf526c6a4245674a2fa6b82dc5e8f69dfba3a9d552/userdata/shm major:0 minor:351 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/070afec86d4588d7b7081c46e8bea71da9940391ce8b8a1f1f332305085ebca7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/070afec86d4588d7b7081c46e8bea71da9940391ce8b8a1f1f332305085ebca7/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a3fe3cd0d75872773797b98a329cbc0c746f6c3f127fbfb1f4c1a7a3a052a95/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a3fe3cd0d75872773797b98a329cbc0c746f6c3f127fbfb1f4c1a7a3a052a95/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0cd0585aea299faded21a33f5383b5092fb1ac9962f5c534580d1feb11a5a58c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0cd0585aea299faded21a33f5383b5092fb1ac9962f5c534580d1feb11a5a58c/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21441c2dab7222e3b168d973f4aed71f970c88c6ea9028620e10dc238645f40f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21441c2dab7222e3b168d973f4aed71f970c88c6ea9028620e10dc238645f40f/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2227b91036d9447bab326f0fffe436b5f507fac3ea16c99420efffa4cc903790/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2227b91036d9447bab326f0fffe436b5f507fac3ea16c99420efffa4cc903790/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22fa6676ded0c91b28b32d11334dff7ae62d8127610991237d29edae9b8b5656/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22fa6676ded0c91b28b32d11334dff7ae62d8127610991237d29edae9b8b5656/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38655434f2520c5e2917c6fae8d88b6f7b7fa718d8f5deeedcf30cef0b89c243/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38655434f2520c5e2917c6fae8d88b6f7b7fa718d8f5deeedcf30cef0b89c243/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e6aa65ba59b277eaf7adf3e04edcd5232a2fee9053f03e8d561441a6c3419cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e6aa65ba59b277eaf7adf3e04edcd5232a2fee9053f03e8d561441a6c3419cc/userdata/shm major:0 minor:279 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f16b832f1a4c5a856e6bf0ca22a73cd23a3e10e3483ed6bd6fe2ff7264d0886/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f16b832f1a4c5a856e6bf0ca22a73cd23a3e10e3483ed6bd6fe2ff7264d0886/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42d007bf2eb83cb4679c66b689dfa99d851c274cfccdd3b74fb8b6be3c440312/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42d007bf2eb83cb4679c66b689dfa99d851c274cfccdd3b74fb8b6be3c440312/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a7aa36d62284423b604b802a6c71bd13958257e15869924b627cd54257f9f71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a7aa36d62284423b604b802a6c71bd13958257e15869924b627cd54257f9f71/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7069e383c5e90e1f48fc0dce4f850b630e915add3755f25704bc7093cbdeae28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7069e383c5e90e1f48fc0dce4f850b630e915add3755f25704bc7093cbdeae28/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76a98cc0e807c17023de22914cec00e315444bfc62406884a5457bc24da527a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76a98cc0e807c17023de22914cec00e315444bfc62406884a5457bc24da527a5/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ea1e10845bd2bf33c03c886f2bd6ed20c988bfc836f634567dd107aa474ac20/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ea1e10845bd2bf33c03c886f2bd6ed20c988bfc836f634567dd107aa474ac20/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b63b00ad000545bb2d57c0037e6d9f1ac050fbf987df19dd240123684ce4be3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b63b00ad000545bb2d57c0037e6d9f1ac050fbf987df19dd240123684ce4be3/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a6991d2ce08040d5e55532dfe0c9cafcc01e539d91fed48c1a9d8073fdbcb46a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a6991d2ce08040d5e55532dfe0c9cafcc01e539d91fed48c1a9d8073fdbcb46a/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7533b85e453b268a4fb61a88978d346221719537f1846cb4d205b1aab1b1451/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7533b85e453b268a4fb61a88978d346221719537f1846cb4d205b1aab1b1451/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aff42e60564e77b412cc0588ce32d9ca1beef086c381cb9deefc67e35867fcea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aff42e60564e77b412cc0588ce32d9ca1beef086c381cb9deefc67e35867fcea/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c36c6a76d6d9bf65bd809f109b63aed4184fc1227bede75fa9d2ba65a9c1b170/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c36c6a76d6d9bf65bd809f109b63aed4184fc1227bede75fa9d2ba65a9c1b170/userdata/shm major:0 minor:320 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4dff5b758ed19f952f60182f5f8fe09284a9a1b0e2c433d522537a9e8e38b26/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4dff5b758ed19f952f60182f5f8fe09284a9a1b0e2c433d522537a9e8e38b26/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5ce2926aaa1b6b38fd605d7bc2906c47d67c146a53ee3e73ff34c8f6862b273/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5ce2926aaa1b6b38fd605d7bc2906c47d67c146a53ee3e73ff34c8f6862b273/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8465dd0f82dfe1e1add0aba8f6155e60ef7df4b52833b40fc8597de1f1faf0f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8465dd0f82dfe1e1add0aba8f6155e60ef7df4b52833b40fc8597de1f1faf0f/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2cf2855e1e18cbff8c7c4f4ff8c2b8b63020f9b606a416adfd2f86609337bb7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2cf2855e1e18cbff8c7c4f4ff8c2b8b63020f9b606a416adfd2f86609337bb7/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f79a5c16d5c71a460bfd24e8ccb80dd3a91b4257e178df5f67daa45b0aa5ef5f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f79a5c16d5c71a460bfd24e8ccb80dd3a91b4257e178df5f67daa45b0aa5ef5f/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12ce6198-3748-49b2-ad3d-f5c3d701fc96/volumes/kubernetes.io~projected/kube-api-access-6bxrc:{mountpoint:/var/lib/kubelet/pods/12ce6198-3748-49b2-ad3d-f5c3d701fc96/volumes/kubernetes.io~projected/kube-api-access-6bxrc major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12ce6198-3748-49b2-ad3d-f5c3d701fc96/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/12ce6198-3748-49b2-ad3d-f5c3d701fc96/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15d37b2c-d051-419e-b0cb-d761cac6ff74/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/15d37b2c-d051-419e-b0cb-d761cac6ff74/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15d37b2c-d051-419e-b0cb-d761cac6ff74/volumes/kubernetes.io~projected/kube-api-access-cdvvx:{mountpoint:/var/lib/kubelet/pods/15d37b2c-d051-419e-b0cb-d761cac6ff74/volumes/kubernetes.io~projected/kube-api-access-cdvvx major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15d37b2c-d051-419e-b0cb-d761cac6ff74/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/15d37b2c-d051-419e-b0cb-d761cac6ff74/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15d37b2c-d051-419e-b0cb-d761cac6ff74/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/15d37b2c-d051-419e-b0cb-d761cac6ff74/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f33677f-ee87-4cd1-bdb0-b74000834db2/volumes/kubernetes.io~projected/kube-api-access-wq6p2:{mountpoint:/var/lib/kubelet/pods/1f33677f-ee87-4cd1-bdb0-b74000834db2/volumes/kubernetes.io~projected/kube-api-access-wq6p2 major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/245acdd3-cebd-49fe-bf1e-867df120ac8e/volumes/kubernetes.io~projected/kube-api-access-76nsp:{mountpoint:/var/lib/kubelet/pods/245acdd3-cebd-49fe-bf1e-867df120ac8e/volumes/kubernetes.io~projected/kube-api-access-76nsp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24a97420-efcd-4a6f-b072-1e5792264188/volumes/kubernetes.io~projected/kube-api-access-8zvs2:{mountpoint:/var/lib/kubelet/pods/24a97420-efcd-4a6f-b072-1e5792264188/volumes/kubernetes.io~projected/kube-api-access-8zvs2 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4/volumes/kubernetes.io~projected/kube-api-access-8l7br:{mountpoint:/var/lib/kubelet/pods/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4/volumes/kubernetes.io~projected/kube-api-access-8l7br major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34ba944d-b672-489d-9acc-fde356202fbe/volumes/kubernetes.io~projected/kube-api-access-6zwts:{mountpoint:/var/lib/kubelet/pods/34ba944d-b672-489d-9acc-fde356202fbe/volumes/kubernetes.io~projected/kube-api-access-6zwts major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20/volumes/kubernetes.io~projected/kube-api-access-gxhlc:{mountpoint:/var/lib/kubelet/pods/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20/volumes/kubernetes.io~projected/kube-api-access-gxhlc major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42d01b00-5ec9-4f8d-a8f2-10e0addd1b67/volumes/kubernetes.io~projected/kube-api-access-fflvq:{mountpoint:/var/lib/kubelet/pods/42d01b00-5ec9-4f8d-a8f2-10e0addd1b67/volumes/kubernetes.io~projected/kube-api-access-fflvq major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42d01b00-5ec9-4f8d-a8f2-10e0addd1b67/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/42d01b00-5ec9-4f8d-a8f2-10e0addd1b67/volumes/kubernetes.io~secret/cert major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/462e871a-1226-4f07-b34c-ef612fa6bf52/volumes/kubernetes.io~projected/kube-api-access-pc6rc:{mountpoint:/var/lib/kubelet/pods/462e871a-1226-4f07-b34c-ef612fa6bf52/volumes/kubernetes.io~projected/kube-api-access-pc6rc major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a6bf982-7a96-41ae-b24d-78cdabbb7c37/volumes/kubernetes.io~projected/kube-api-access-sj7td:{mountpoint:/var/lib/kubelet/pods/4a6bf982-7a96-41ae-b24d-78cdabbb7c37/volumes/kubernetes.io~projected/kube-api-access-sj7td major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/510fbc36-62de-4096-b8e8-d8606a4ad7d5/volumes/kubernetes.io~projected/kube-api-access-fhh48:{mountpoint:/var/lib/kubelet/pods/510fbc36-62de-4096-b8e8-d8606a4ad7d5/volumes/kubernetes.io~projected/kube-api-access-fhh48 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/510fbc36-62de-4096-b8e8-d8606a4ad7d5/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/510fbc36-62de-4096-b8e8-d8606a4ad7d5/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/542dd56a-cfab-40e2-a01b-a172651d46f8/volumes/kubernetes.io~projected/kube-api-access-cvp8m:{mountpoint:/var/lib/kubelet/pods/542dd56a-cfab-40e2-a01b-a172651d46f8/volumes/kubernetes.io~projected/kube-api-access-cvp8m major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ea4f6f2-4699-46d7-91bc-8391dde191eb/volumes/kubernetes.io~projected/kube-api-access-gkv6p:{mountpoint:/var/lib/kubelet/pods/5ea4f6f2-4699-46d7-91bc-8391dde191eb/volumes/kubernetes.io~projected/kube-api-access-gkv6p major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/793c02dd-468b-4308-bac2-67d04e40579c/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/793c02dd-468b-4308-bac2-67d04e40579c/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/793c02dd-468b-4308-bac2-67d04e40579c/volumes/kubernetes.io~projected/kube-api-access-5cjnm:{mountpoint:/var/lib/kubelet/pods/793c02dd-468b-4308-bac2-67d04e40579c/volumes/kubernetes.io~projected/kube-api-access-5cjnm major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/793c02dd-468b-4308-bac2-67d04e40579c/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/793c02dd-468b-4308-bac2-67d04e40579c/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/793c02dd-468b-4308-bac2-67d04e40579c/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/793c02dd-468b-4308-bac2-67d04e40579c/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84556453-e5d9-4449-ba59-819c248e1e2a/volumes/kubernetes.io~projected/kube-api-access-4vqts:{mountpoint:/var/lib/kubelet/pods/84556453-e5d9-4449-ba59-819c248e1e2a/volumes/kubernetes.io~projected/kube-api-access-4vqts major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d21a1ae-f712-4f89-b34f-8016fd918484/volumes/kubernetes.io~projected/kube-api-access-2hd42:{mountpoint:/var/lib/kubelet/pods/8d21a1ae-f712-4f89-b34f-8016fd918484/volumes/kubernetes.io~projected/kube-api-access-2hd42 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d21a1ae-f712-4f89-b34f-8016fd918484/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8d21a1ae-f712-4f89-b34f-8016fd918484/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e456784-7491-426a-80e5-17a9dfa2f07f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/8e456784-7491-426a-80e5-17a9dfa2f07f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e456784-7491-426a-80e5-17a9dfa2f07f/volumes/kubernetes.io~projected/kube-api-access-v7n4p:{mountpoint:/var/lib/kubelet/pods/8e456784-7491-426a-80e5-17a9dfa2f07f/volumes/kubernetes.io~projected/kube-api-access-v7n4p major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e456784-7491-426a-80e5-17a9dfa2f07f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/8e456784-7491-426a-80e5-17a9dfa2f07f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f39698b-0204-4492-ac16-e1728e71b8d6/volumes/kubernetes.io~projected/kube-api-access-8nd4m:{mountpoint:/var/lib/kubelet/pods/8f39698b-0204-4492-ac16-e1728e71b8d6/volumes/kubernetes.io~projected/kube-api-access-8nd4m major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af51f192-4170-4d3b-85ae-7d2ceb14de05/volumes/kubernetes.io~projected/kube-api-access-rd729:{mountpoint:/var/lib/kubelet/pods/af51f192-4170-4d3b-85ae-7d2ceb14de05/volumes/kubernetes.io~projected/kube-api-access-rd729 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7559e5a-a8fd-450c-8f40-f2072e93a9b5/volumes/kubernetes.io~projected/kube-api-access-t42hf:{mountpoint:/var/lib/kubelet/pods/c7559e5a-a8fd-450c-8f40-f2072e93a9b5/volumes/kubernetes.io~projected/kube-api-access-t42hf major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf6a6e00-44f7-4af0-be4d-3650c552d729/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/cf6a6e00-44f7-4af0-be4d-3650c552d729/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0d6aa7f-3a2b-482f-8339-291a11a77c22/volumes/kubernetes.io~projected/kube-api-access-brl7b:{mountpoint:/var/lib/kubelet/pods/d0d6aa7f-3a2b-482f-8339-291a11a77c22/volumes/kubernetes.io~projected/kube-api-access-brl7b major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2af25ca-9ad2-481c-addd-ebe3d999fc4e/volumes/kubernetes.io~projected/kube-api-access-fhvdt:{mountpoint:/var/lib/kubelet/pods/d2af25ca-9ad2-481c-addd-ebe3d999fc4e/volumes/kubernetes.io~projected/kube-api-access-fhvdt major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e29718e2-f954-4ec8-9db2-d6c3638bc51c/volumes/kubernetes.io~projected/kube-api-access-6mzl9:{mountpoint:/var/lib/kubelet/pods/e29718e2-f954-4ec8-9db2-d6c3638bc51c/volumes/kubernetes.io~projected/kube-api-access-6mzl9 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e29718e2-f954-4ec8-9db2-d6c3638bc51c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e29718e2-f954-4ec8-9db2-d6c3638bc51c/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3c555a1-e6bd-4042-be58-404cb6693922/volumes/kubernetes.io~projected/kube-api-access-ltgln:{mountpoint:/var/lib/kubelet/pods/e3c555a1-e6bd-4042-be58-404cb6693922/volumes/kubernetes.io~projected/kube-api-access-ltgln major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3c555a1-e6bd-4042-be58-404cb6693922/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e3c555a1-e6bd-4042-be58-404cb6693922/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e863782c-c289-4fa5-93b1-b475358a5001/volumes/kubernetes.io~projected/kube-api-access-l9dm2:{mountpoint:/var/lib/kubelet/pods/e863782c-c289-4fa5-93b1-b475358a5001/volumes/kubernetes.io~projected/kube-api-access-l9dm2 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eafb0c22-9550-4b1c-8134-fc80562025eb/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/eafb0c22-9550-4b1c-8134-fc80562025eb/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eafb0c22-9550-4b1c-8134-fc80562025eb/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/eafb0c22-9550-4b1c-8134-fc80562025eb/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eafb0c22-9550-4b1c-8134-fc80562025eb/volumes/kubernetes.io~projected/kube-api-access-2mzjv:{mountpoint:/var/lib/kubelet/pods/eafb0c22-9550-4b1c-8134-fc80562025eb/volumes/kubernetes.io~projected/kube-api-access-2mzjv major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb7b7d50-f5bf-45b4-acaa-f1bc19831287/volumes/kubernetes.io~projected/kube-api-access-5m4g8:{mountpoint:/var/lib/kubelet/pods/eb7b7d50-f5bf-45b4-acaa-f1bc19831287/volumes/kubernetes.io~projected/kube-api-access-5m4g8 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb7b7d50-f5bf-45b4-acaa-f1bc19831287/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/eb7b7d50-f5bf-45b4-acaa-f1bc19831287/volumes/kubernetes.io~secret/default-certificate major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb7b7d50-f5bf-45b4-acaa-f1bc19831287/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/eb7b7d50-f5bf-45b4-acaa-f1bc19831287/volumes/kubernetes.io~secret/stats-auth major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed5e1db5-564c-4abd-87c4-a2dda264a926/volumes/kubernetes.io~projected/kube-api-access-fbvs7:{mountpoint:/var/lib/kubelet/pods/ed5e1db5-564c-4abd-87c4-a2dda264a926/volumes/kubernetes.io~projected/kube-api-access-fbvs7 major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb/volumes/kubernetes.io~projected/kube-api-access-dbfzh:{mountpoint:/var/lib/kubelet/pods/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb/volumes/kubernetes.io~projected/kube-api-access-dbfzh major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff7262fd-b29d-4c72-b717-ef9742bdd2db/volumes/kubernetes.io~projected/kube-api-access-8nwhb:{mountpoint:/var/lib/kubelet/pods/ff7262fd-b29d-4c72-b717-ef9742bdd2db/volumes/kubernetes.io~projected/kube-api-access-8nwhb major:0 minor:236 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/286c2490fadc745794131b1d5b88700953e1d9a06479a438267ac89c7499ed1b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/747d8b8aeba9d52fc1f03c39d65ef9654a6a9b3e69f48c95e313039557d49632/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/8e9c5e77e171c781a64924fee5eee304deb2e8578ac0d46d1310f380fdd9fe6c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/14128f767f387db3b38558b21ed0942b242fa00bc661f225be50f186ec5a4547/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b5424a6b5d08841a9c418c52cb003a193620a87fb02b733ddf6b444de67e8da9/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/604452bec3c8fc7176f0ea8d6d0f225073da9b6e21362d73bf12acbbd7fe08e6/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/ca193199a8766767d3881555d0ed33686ebfa84fb1c6b8c0a7c0f16e8df9eec1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/7c081111244d024fc4b895d1da3b985310ae2929ac2f21fa8f17786a3c87194b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/a8070c7d8946a95ebca523f300e7f25a729943ad363db0330470155a9b3f38cb/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/2a2d0268e28d5800ad987ae937eb91ec396395e9b9bbe5d21f305b431ab65762/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/5e737ec4065839384fcd026d8ccb7da016bd977d753b43c5e8542a5743b8bcf0/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/b9ec7add02a0e3e798a3046baa3c4ee143caf34b176acd58bf14e693112dd74e/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/2a9d403d121dffd130b0003401ec7debaff3e98abbbee24efb1633c188fc6741/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/4d8e0277b53410c5753e2053e411c692172dbd6564302f157fc98d0c14ef2b0a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/0c63c0c4146782cbc0b9fa7d2cc98d154bb7a4cfdf7bdc807332d5078c6eafec/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/5b347624670166010042bf9bc55281afa7bc830d2132adc0c14b21e123a81b5f/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/70a046cf3ab5800a46ddc678fac7380eb071dd8eb8845ca91234ecdabd467b5c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/ebb2685ff00e504d3a94a01fa47afcff6b5b39ed4eca2b21f211e5ad18c625ae/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ac3ddf3fa3c74fcd57ea7e8c2ca4dea97f4bf9004d6ac2088ab550d163a3e021/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/4d42433642abf41cb2799337471c0cd081403cb39b50a3c3067a953cffc58ee0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/4d57a0b79bff97ec2a3bda6183cad1d54ec50d1a97cd3e549b8e2e12a322ffdc/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/1092e2ce1c5e8d1541f7fbedc4bd833b6f1f3d0b85d33adee7488c70ae1844a9/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/28223941c302a801c938d38fde5c187023d1ed19f455fa0523c7993e7180a455/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/b9f04c4fcb3334bb1715fdd97acb2556113ace760eee6dbda5a680141c847616/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/1fb7cc41b61c27a8abebd75445d5b2183797f457979da859c24aa171b3a79f10/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/a6bb844fcbf45ea1bded9f29fce34c30612a9da6c73e9d5f7d157517a298191b/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/183568afbd74f7190369b306d5972f5b71ead5f3701685858902ed4e1f974d17/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/811eb6ff904393dd3d6de9b7f356aa9875117f0c55800188cb9963dd5b02b77f/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/c7a493e5ccfb5c505fdd0a39bbf000e5c8acc55f18789ec6ab1ae781749d5394/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/9d2180cda10ebb07ec5900dd53392548161bd9282ce721e42c85a4ca743fbb46/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/8a197262782449e13d6281c874cbb9b628a65452c9c6ece413c4180913f6b6b3/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/ec00bd6580f8770802dca9663ccb2e6a55e50fb117b45c634528bf1e34b1f0b3/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/19652e9aa5736307d2bb4b127d6a1bda73f37907bec22a308494bb2d66fe6f59/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/ee16b7561b948bf6188aacdcbd1262e78db1e336aa1cc9ff14839d1287e9a0b5/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/94abb0b3788acc7cc81e7274275f9d81e40118f1ff935d9b717870a504bbde97/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/1a304a8d335c2ae9d82cbe174cebc747554608a10bdd20802a040c386076bc04/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/30cfa5a9cab768ec6a5603c8eb1c57adf33e6a2a0ed2158fe7b3f14004fdf974/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/c145094e45cb7b2adcb96ef0eabe1d618f831bf0aeeb78c71b24518f09894d47/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/05f1ad4b144412f13f93a91a5cc1d8092a27345b303a90ef677dde1118aba58e/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/e5f4e26dd9c7a7506bbb2bac46d6fbcdff93ca83eedb652dab9538d9c31673e7/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/b6d5f9bea5cbc79b70280727704c1b86ccb442ca15b21ff4cae58e0dbf3493e2/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/a946539f3ddd3a0fd6bde49f5d513efbb0b7c5e7398724c6860886d9766468e0/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/ece6e7ef269c3025f82be24242f50d748eb27fa181ff38407b6313581eaef5e9/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/be3d91f22c7560e99d238dd9bbcb4a9bb925c20c686eeae1cdeabb8c9024414e/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/e54a1028883754557818577b3f4bb2586f0f995a964b66358ca1eab463c94929/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/768a2320f0f0240949544518ea9b80d243939901ad8fd87268c50024335f00f1/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/dc9791f879fa2612b64fc1b62a85bf0da03da941243b42b0b4d88669beb3f7e7/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/ad7267d1ac91620b5b14561f95bc7c098c0151e365fcc22d02e52431d6bc3d36/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/0fa174ca8695fa2a3590f9ff2b0e14c3d4dfb1e6f4553b812f9d7c66a1d775fe/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/4c9d1f01cc6c960e8324f8148a4f0f5c5fc40dbeceeecf5c39cd545550c55719/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/6b10c51d39e34d0f90bf41cc90db56ab8194aeed01e2e5f02334a7b7628a2c06/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/72440de59db07b4df54f0e01755e1bc533aab58752f1253e94687ed5feb7d3a1/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/0f7a62843dd2f24e6ea079c2fa2bbbc87449b0fe6cd68a3826a35aa63157e683/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/4e22bf0118e81469b6e9e278df024d45b5ec5fc5f84c5ef42dbc33729380fe3f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/875f11efe5fd4f889d2e5b0d8d7e294bc98580de29b5ec5f2d11487c73441e76/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/8d69bd7c91541fa806e1baac72cae01d29272f877ffa7997f15a972f162adbae/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/1d6430e0bb86a8271a1d447141e242f7ff7c362223478a1efea10590d38d5b76/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/c83fbd2a38f658282b742cbab563872037af5b42c8dca9fe70eb8df645044880/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 03 20:47:30.903485 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.902576 4963 manager.go:217] Machine: {Timestamp:2026-11-03 20:47:30.902076143 +0000 UTC m=+0.062148405 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3134324 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25084211570d2077fa2d96f8dd88a2 SystemUUID:ec250842-1157-0d20-77fa-2d96f8dd88a2 BootID:f4d3bae6-cb22-4a2b-aa62-2c6d6f9110bf Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2227b91036d9447bab326f0fffe436b5f507fac3ea16c99420efffa4cc903790/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0d6aa7f-3a2b-482f-8339-291a11a77c22/volumes/kubernetes.io~projected/kube-api-access-brl7b DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/eafb0c22-9550-4b1c-8134-fc80562025eb/volumes/kubernetes.io~projected/kube-api-access-2mzjv DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/34ba944d-b672-489d-9acc-fde356202fbe/volumes/kubernetes.io~projected/kube-api-access-6zwts DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/15d37b2c-d051-419e-b0cb-d761cac6ff74/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:235 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb/volumes/kubernetes.io~projected/kube-api-access-dbfzh DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8f39698b-0204-4492-ac16-e1728e71b8d6/volumes/kubernetes.io~projected/kube-api-access-8nd4m DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/15d37b2c-d051-419e-b0cb-d761cac6ff74/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e456784-7491-426a-80e5-17a9dfa2f07f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/793c02dd-468b-4308-bac2-67d04e40579c/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c36c6a76d6d9bf65bd809f109b63aed4184fc1227bede75fa9d2ba65a9c1b170/userdata/shm DeviceMajor:0 DeviceMinor:320 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7559e5a-a8fd-450c-8f40-f2072e93a9b5/volumes/kubernetes.io~projected/kube-api-access-t42hf DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22fa6676ded0c91b28b32d11334dff7ae62d8127610991237d29edae9b8b5656/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/510fbc36-62de-4096-b8e8-d8606a4ad7d5/volumes/kubernetes.io~projected/kube-api-access-fhh48 DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03ec5291801f3033463914cf526c6a4245674a2fa6b82dc5e8f69dfba3a9d552/userdata/shm DeviceMajor:0 DeviceMinor:351 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21441c2dab7222e3b168d973f4aed71f970c88c6ea9028620e10dc238645f40f/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/510fbc36-62de-4096-b8e8-d8606a4ad7d5/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7069e383c5e90e1f48fc0dce4f850b630e915add3755f25704bc7093cbdeae28/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4dff5b758ed19f952f60182f5f8fe09284a9a1b0e2c433d522537a9e8e38b26/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38655434f2520c5e2917c6fae8d88b6f7b7fa718d8f5deeedcf30cef0b89c243/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb7b7d50-f5bf-45b4-acaa-f1bc19831287/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/42d01b00-5ec9-4f8d-a8f2-10e0addd1b67/volumes/kubernetes.io~projected/kube-api-access-fflvq DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b63b00ad000545bb2d57c0037e6d9f1ac050fbf987df19dd240123684ce4be3/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0cd0585aea299faded21a33f5383b5092fb1ac9962f5c534580d1feb11a5a58c/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4/volumes/kubernetes.io~projected/kube-api-access-8l7br DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/12ce6198-3748-49b2-ad3d-f5c3d701fc96/volumes/kubernetes.io~projected/kube-api-access-6bxrc DeviceMajor:0 DeviceMinor:363 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/84556453-e5d9-4449-ba59-819c248e1e2a/volumes/kubernetes.io~projected/kube-api-access-4vqts DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42d01b00-5ec9-4f8d-a8f2-10e0addd1b67/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f16b832f1a4c5a856e6bf0ca22a73cd23a3e10e3483ed6bd6fe2ff7264d0886/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cf6a6e00-44f7-4af0-be4d-3650c552d729/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8465dd0f82dfe1e1add0aba8f6155e60ef7df4b52833b40fc8597de1f1faf0f/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/245acdd3-cebd-49fe-bf1e-867df120ac8e/volumes/kubernetes.io~projected/kube-api-access-76nsp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a6991d2ce08040d5e55532dfe0c9cafcc01e539d91fed48c1a9d8073fdbcb46a/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ea1e10845bd2bf33c03c886f2bd6ed20c988bfc836f634567dd107aa474ac20/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8d21a1ae-f712-4f89-b34f-8016fd918484/volumes/kubernetes.io~projected/kube-api-access-2hd42 DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e29718e2-f954-4ec8-9db2-d6c3638bc51c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e6aa65ba59b277eaf7adf3e04edcd5232a2fee9053f03e8d561441a6c3419cc/userdata/shm DeviceMajor:0 DeviceMinor:279 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/070afec86d4588d7b7081c46e8bea71da9940391ce8b8a1f1f332305085ebca7/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8e456784-7491-426a-80e5-17a9dfa2f07f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eafb0c22-9550-4b1c-8134-fc80562025eb/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/542dd56a-cfab-40e2-a01b-a172651d46f8/volumes/kubernetes.io~projected/kube-api-access-cvp8m DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/15d37b2c-d051-419e-b0cb-d761cac6ff74/volumes/kubernetes.io~projected/kube-api-access-cdvvx DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a3fe3cd0d75872773797b98a329cbc0c746f6c3f127fbfb1f4c1a7a3a052a95/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/462e871a-1226-4f07-b34c-ef612fa6bf52/volumes/kubernetes.io~projected/kube-api-access-pc6rc DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8d21a1ae-f712-4f89-b34f-8016fd918484/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d2af25ca-9ad2-481c-addd-ebe3d999fc4e/volumes/kubernetes.io~projected/kube-api-access-fhvdt DeviceMajor:0 DeviceMinor:346 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e29718e2-f954-4ec8-9db2-d6c3638bc51c/volumes/kubernetes.io~projected/kube-api-access-6mzl9 DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/793c02dd-468b-4308-bac2-67d04e40579c/volumes/kubernetes.io~projected/kube-api-access-5cjnm DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/15d37b2c-d051-419e-b0cb-d761cac6ff74/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/12ce6198-3748-49b2-ad3d-f5c3d701fc96/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8e456784-7491-426a-80e5-17a9dfa2f07f/volumes/kubernetes.io~projected/kube-api-access-v7n4p DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e863782c-c289-4fa5-93b1-b475358a5001/volumes/kubernetes.io~projected/kube-api-access-l9dm2 DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5ce2926aaa1b6b38fd605d7bc2906c47d67c146a53ee3e73ff34c8f6862b273/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/793c02dd-468b-4308-bac2-67d04e40579c/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2cf2855e1e18cbff8c7c4f4ff8c2b8b63020f9b606a416adfd2f86609337bb7/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a7aa36d62284423b604b802a6c71bd13958257e15869924b627cd54257f9f71/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3c555a1-e6bd-4042-be58-404cb6693922/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f79a5c16d5c71a460bfd24e8ccb80dd3a91b4257e178df5f67daa45b0aa5ef5f/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7533b85e453b268a4fb61a88978d346221719537f1846cb4d205b1aab1b1451/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af51f192-4170-4d3b-85ae-7d2ceb14de05/volumes/kubernetes.io~projected/kube-api-access-rd729 DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4a6bf982-7a96-41ae-b24d-78cdabbb7c37/volumes/kubernetes.io~projected/kube-api-access-sj7td DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ea4f6f2-4699-46d7-91bc-8391dde191eb/volumes/kubernetes.io~projected/kube-api-access-gkv6p DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:276 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20/volumes/kubernetes.io~projected/kube-api-access-gxhlc DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eb7b7d50-f5bf-45b4-acaa-f1bc19831287/volumes/kubernetes.io~projected/kube-api-access-5m4g8 DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ff7262fd-b29d-4c72-b717-ef9742bdd2db/volumes/kubernetes.io~projected/kube-api-access-8nwhb DeviceMajor:0 DeviceMinor:236 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/793c02dd-468b-4308-bac2-67d04e40579c/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42d007bf2eb83cb4679c66b689dfa99d851c274cfccdd3b74fb8b6be3c440312/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76a98cc0e807c17023de22914cec00e315444bfc62406884a5457bc24da527a5/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eafb0c22-9550-4b1c-8134-fc80562025eb/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24a97420-efcd-4a6f-b072-1e5792264188/volumes/kubernetes.io~projected/kube-api-access-8zvs2 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3c555a1-e6bd-4042-be58-404cb6693922/volumes/kubernetes.io~projected/kube-api-access-ltgln DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eb7b7d50-f5bf-45b4-acaa-f1bc19831287/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aff42e60564e77b412cc0588ce32d9ca1beef086c381cb9deefc67e35867fcea/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1f33677f-ee87-4cd1-bdb0-b74000834db2/volumes/kubernetes.io~projected/kube-api-access-wq6p2 DeviceMajor:0 DeviceMinor:319 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed5e1db5-564c-4abd-87c4-a2dda264a926/volumes/kubernetes.io~projected/kube-api-access-fbvs7 DeviceMajor:0 DeviceMinor:353 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:03ec5291801f303 MacAddress:06:a9:91:20:32:b9 Speed:10000 Mtu:8901} {Name:070afec86d4588d MacAddress:a6:0e:cf:3b:f7:a3 Speed:10000 Mtu:8901} {Name:0cd0585aea299fa MacAddress:ae:08:39:7f:32:ed Speed:10000 Mtu:8901} {Name:21441c2dab7222e MacAddress:02:33:3f:12:fe:d7 Speed:10000 Mtu:8901} {Name:3e6aa65ba59b277 MacAddress:c2:3c:34:1c:b5:cf Speed:10000 Mtu:8901} {Name:3f16b832f1a4c5a MacAddress:2e:56:5c:21:f8:25 Speed:10000 Mtu:8901} {Name:42d007bf2eb83cb MacAddress:6e:48:55:00:39:1f Speed:10000 Mtu:8901} {Name:76a98cc0e807c17 MacAddress:06:0f:6a:ca:1d:11 Speed:10000 Mtu:8901} {Name:8b63b00ad000545 MacAddress:ea:61:c4:df:e7:01 Speed:10000 Mtu:8901} {Name:a7533b85e453b26 MacAddress:ca:b0:4f:98:83:32 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:c2:30:11:33 Speed:0 Mtu:9001} {Name:br-int MacAddress:d6:d0:e3:4e:89:3f Speed:0 Mtu:8901} {Name:c36c6a76d6d9bf6 MacAddress:02:38:1a:d4:8d:1f Speed:10000 Mtu:8901} {Name:c4dff5b758ed19f MacAddress:7e:83:84:22:51:89 Speed:10000 Mtu:8901} {Name:c5ce2926aaa1b6b MacAddress:36:79:b5:f4:c1:5c Speed:10000 Mtu:8901} {Name:c8465dd0f82dfe1 MacAddress:22:61:6c:20:35:91 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:c2:30:11:33 Speed:0 Mtu:9001} {Name:f79a5c16d5c71a4 MacAddress:ea:09:97:61:a1:48 Speed:10000 Mtu:8901} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:72:91:ff:5c:68:09 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 20:47:30.903485 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.903006 4963 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 20:47:30.903485 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.903055 4963 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 20:47:30.903880 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.903502 4963 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 20:47:30.903880 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.903539 4963 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-134.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 20:47:30.903880 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.903737 4963 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 20:47:30.903880 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.903749 4963 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 20:47:30.903880 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.903764 4963 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:47:30.903880 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.903791 4963 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:47:30.903880 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.903818 4963 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 20:47:30.904217 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.904100 4963 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 20:47:30.904271 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.904251 4963 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 20:47:30.904317 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.904301 4963 kubelet.go:493] "Attempting to sync node with API server" Nov 03 20:47:30.904368 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.904316 4963 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 20:47:30.904368 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.904333 4963 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 20:47:30.904368 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.904348 4963 kubelet.go:405] "Adding apiserver pod source" Nov 03 20:47:30.904368 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.904360 4963 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 20:47:30.905288 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.905270 4963 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 20:47:30.905288 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.905285 4963 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:47:30.905954 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.905928 4963 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 20:47:30.920344 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.919531 4963 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 20:47:30.920344 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.919559 4963 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:47:30.920344 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.919740 4963 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 20:47:30.920344 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.920163 4963 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 20:47:30.920344 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.920207 4963 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 20:47:30.920596 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.920383 4963 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 20:47:30.920596 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.920402 4963 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 20:47:30.920596 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.920412 4963 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 20:47:30.920596 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.920421 4963 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 20:47:30.920596 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.920461 4963 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 20:47:30.920596 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.920474 4963 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 20:47:30.920596 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.920481 4963 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 20:47:30.920596 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.920493 4963 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 20:47:30.920596 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.920508 4963 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 20:47:30.920596 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.920518 4963 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 20:47:30.920596 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.920531 4963 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 20:47:30.920596 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.920568 4963 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 20:47:30.920596 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.920601 4963 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 20:47:30.921055 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.920608 4963 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 20:47:30.924567 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.924548 4963 server.go:1263] "Started kubelet" Nov 03 20:47:30.925176 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.924640 4963 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 20:47:30.925176 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.924681 4963 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 20:47:30.925176 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.924778 4963 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 20:47:30.925373 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.925289 4963 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 20:47:30.925187 ip-10-0-130-134 systemd[1]: Started Kubernetes Kubelet. Nov 03 20:47:30.926969 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.926064 4963 server.go:317] "Adding debug handlers to kubelet server" Nov 03 20:47:30.929062 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.928540 4963 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 20:47:30.929062 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.928638 4963 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 20:47:30.929062 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.928651 4963 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 20:47:30.929062 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:30.928666 4963 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-134.us-west-2.compute.internal\" not found" Nov 03 20:47:30.929062 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.928539 4963 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 20:47:30.929062 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.928736 4963 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 20:47:30.929062 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.928803 4963 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:41:49 +0000 UTC" deadline="2028-04-19 03:46:23.018365319 +0000 UTC" Nov 03 20:47:30.929062 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.928838 4963 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12774h58m52.089530114s" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.933277 4963 factory.go:55] Registering systemd factory Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.933304 4963 factory.go:223] Registration of the systemd container factory successfully Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.933728 4963 factory.go:153] Registering CRI-O factory Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.933747 4963 factory.go:223] Registration of the crio container factory successfully Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.933810 4963 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.933830 4963 factory.go:103] Registering Raw factory Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.933849 4963 manager.go:1199] Started watching for new ooms in manager Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934122 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15d37b2c-d051-419e-b0cb-d761cac6ff74" volumeName="kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-image-registry-private-configuration" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934179 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="793c02dd-468b-4308-bac2-67d04e40579c" volumeName="kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-trusted-ca" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934192 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="793c02dd-468b-4308-bac2-67d04e40579c" volumeName="kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-bound-sa-token" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934228 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb7b7d50-f5bf-45b4-acaa-f1bc19831287" volumeName="kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-stats-auth" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934241 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12ce6198-3748-49b2-ad3d-f5c3d701fc96" volumeName="kubernetes.io/projected/12ce6198-3748-49b2-ad3d-f5c3d701fc96-kube-api-access-6bxrc" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934255 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e456784-7491-426a-80e5-17a9dfa2f07f" volumeName="kubernetes.io/configmap/8e456784-7491-426a-80e5-17a9dfa2f07f-ovnkube-script-lib" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934287 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eafb0c22-9550-4b1c-8134-fc80562025eb" volumeName="kubernetes.io/empty-dir/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-tuned" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934323 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a6bf982-7a96-41ae-b24d-78cdabbb7c37" volumeName="kubernetes.io/projected/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-kube-api-access-sj7td" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934348 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d21a1ae-f712-4f89-b34f-8016fd918484" volumeName="kubernetes.io/secret/8d21a1ae-f712-4f89-b34f-8016fd918484-serving-cert" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934447 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e456784-7491-426a-80e5-17a9dfa2f07f" volumeName="kubernetes.io/configmap/8e456784-7491-426a-80e5-17a9dfa2f07f-ovnkube-config" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934480 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29718e2-f954-4ec8-9db2-d6c3638bc51c" volumeName="kubernetes.io/configmap/e29718e2-f954-4ec8-9db2-d6c3638bc51c-config" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934507 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3c555a1-e6bd-4042-be58-404cb6693922" volumeName="kubernetes.io/configmap/e3c555a1-e6bd-4042-be58-404cb6693922-config" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934550 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1eb3e05-8b22-4992-97d9-c71ecc6b99dc" volumeName="kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934567 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb02f36e-58c6-47fb-b6f7-0ba78d029ffb" volumeName="kubernetes.io/configmap/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934612 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f33677f-ee87-4cd1-bdb0-b74000834db2" volumeName="kubernetes.io/empty-dir/1f33677f-ee87-4cd1-bdb0-b74000834db2-archives-path" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934682 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ea105ed-0fa7-46a4-8d9f-94714d3a9b20" volumeName="kubernetes.io/configmap/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-cni-binary-copy" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934696 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d21a1ae-f712-4f89-b34f-8016fd918484" volumeName="kubernetes.io/configmap/8d21a1ae-f712-4f89-b34f-8016fd918484-trusted-ca-bundle" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934720 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e456784-7491-426a-80e5-17a9dfa2f07f" volumeName="kubernetes.io/configmap/8e456784-7491-426a-80e5-17a9dfa2f07f-env-overrides" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934733 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15d37b2c-d051-419e-b0cb-d761cac6ff74" volumeName="kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-certificates" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934745 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15d37b2c-d051-419e-b0cb-d761cac6ff74" volumeName="kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-kube-api-access-cdvvx" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934762 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2955ecad-cad6-48d2-a2f0-1c1aae3d47f4" volumeName="kubernetes.io/empty-dir/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-data-volume" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934776 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84556453-e5d9-4449-ba59-819c248e1e2a" volumeName="kubernetes.io/projected/84556453-e5d9-4449-ba59-819c248e1e2a-kube-api-access-4vqts" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934792 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f39698b-0204-4492-ac16-e1728e71b8d6" volumeName="kubernetes.io/projected/8f39698b-0204-4492-ac16-e1728e71b8d6-kube-api-access-8nd4m" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934804 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eafb0c22-9550-4b1c-8134-fc80562025eb" volumeName="kubernetes.io/empty-dir/eafb0c22-9550-4b1c-8134-fc80562025eb-tmp" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934823 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="462e871a-1226-4f07-b34c-ef612fa6bf52" volumeName="kubernetes.io/projected/462e871a-1226-4f07-b34c-ef612fa6bf52-kube-api-access-pc6rc" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934839 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="542dd56a-cfab-40e2-a01b-a172651d46f8" volumeName="kubernetes.io/projected/542dd56a-cfab-40e2-a01b-a172651d46f8-kube-api-access-cvp8m" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934853 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="793c02dd-468b-4308-bac2-67d04e40579c" volumeName="kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-image-registry-private-configuration" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934875 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="793c02dd-468b-4308-bac2-67d04e40579c" volumeName="kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-installation-pull-secrets" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934931 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e456784-7491-426a-80e5-17a9dfa2f07f" volumeName="kubernetes.io/secret/8e456784-7491-426a-80e5-17a9dfa2f07f-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934948 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42d01b00-5ec9-4f8d-a8f2-10e0addd1b67" volumeName="kubernetes.io/projected/42d01b00-5ec9-4f8d-a8f2-10e0addd1b67-kube-api-access-fflvq" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934961 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3c555a1-e6bd-4042-be58-404cb6693922" volumeName="kubernetes.io/secret/e3c555a1-e6bd-4042-be58-404cb6693922-serving-cert" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934977 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb7b7d50-f5bf-45b4-acaa-f1bc19831287" volumeName="kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-default-certificate" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934993 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15d37b2c-d051-419e-b0cb-d761cac6ff74" volumeName="kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-trusted-ca" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935014 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f33677f-ee87-4cd1-bdb0-b74000834db2" volumeName="kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-service-ca-bundle" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935026 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="245acdd3-cebd-49fe-bf1e-867df120ac8e" volumeName="kubernetes.io/projected/245acdd3-cebd-49fe-bf1e-867df120ac8e-kube-api-access-76nsp" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935038 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29718e2-f954-4ec8-9db2-d6c3638bc51c" volumeName="kubernetes.io/secret/e29718e2-f954-4ec8-9db2-d6c3638bc51c-serving-cert" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935056 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb96b430-8876-41db-a35c-19db3cee7635" volumeName="kubernetes.io/configmap/fb96b430-8876-41db-a35c-19db3cee7635-nginx-conf" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935068 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15d37b2c-d051-419e-b0cb-d761cac6ff74" volumeName="kubernetes.io/empty-dir/15d37b2c-d051-419e-b0cb-d761cac6ff74-ca-trust-extracted" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935087 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ea105ed-0fa7-46a4-8d9f-94714d3a9b20" volumeName="kubernetes.io/configmap/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-daemon-config" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935101 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="510fbc36-62de-4096-b8e8-d8606a4ad7d5" volumeName="kubernetes.io/secret/510fbc36-62de-4096-b8e8-d8606a4ad7d5-serving-cert" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935113 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="793c02dd-468b-4308-bac2-67d04e40579c" volumeName="kubernetes.io/empty-dir/793c02dd-468b-4308-bac2-67d04e40579c-ca-trust-extracted" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935128 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb02f36e-58c6-47fb-b6f7-0ba78d029ffb" volumeName="kubernetes.io/projected/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-kube-api-access-dbfzh" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935160 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff7262fd-b29d-4c72-b717-ef9742bdd2db" volumeName="kubernetes.io/configmap/ff7262fd-b29d-4c72-b717-ef9742bdd2db-config-volume" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935189 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ea105ed-0fa7-46a4-8d9f-94714d3a9b20" volumeName="kubernetes.io/projected/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-kube-api-access-gxhlc" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935200 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="542dd56a-cfab-40e2-a01b-a172651d46f8" volumeName="kubernetes.io/configmap/542dd56a-cfab-40e2-a01b-a172651d46f8-iptables-alerter-script" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935212 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e456784-7491-426a-80e5-17a9dfa2f07f" volumeName="kubernetes.io/projected/8e456784-7491-426a-80e5-17a9dfa2f07f-kube-api-access-v7n4p" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935228 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7559e5a-a8fd-450c-8f40-f2072e93a9b5" volumeName="kubernetes.io/projected/c7559e5a-a8fd-450c-8f40-f2072e93a9b5-kube-api-access-t42hf" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935244 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0d6aa7f-3a2b-482f-8339-291a11a77c22" volumeName="kubernetes.io/projected/d0d6aa7f-3a2b-482f-8339-291a11a77c22-kube-api-access-brl7b" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935262 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb02f36e-58c6-47fb-b6f7-0ba78d029ffb" volumeName="kubernetes.io/configmap/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-cni-binary-copy" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935276 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12ce6198-3748-49b2-ad3d-f5c3d701fc96" volumeName="kubernetes.io/configmap/12ce6198-3748-49b2-ad3d-f5c3d701fc96-signing-cabundle" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935310 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15d37b2c-d051-419e-b0cb-d761cac6ff74" volumeName="kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-bound-sa-token" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935324 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15d37b2c-d051-419e-b0cb-d761cac6ff74" volumeName="kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-installation-pull-secrets" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935343 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d21a1ae-f712-4f89-b34f-8016fd918484" volumeName="kubernetes.io/configmap/8d21a1ae-f712-4f89-b34f-8016fd918484-service-ca-bundle" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935360 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d21a1ae-f712-4f89-b34f-8016fd918484" volumeName="kubernetes.io/projected/8d21a1ae-f712-4f89-b34f-8016fd918484-kube-api-access-2hd42" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935374 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf6a6e00-44f7-4af0-be4d-3650c552d729" volumeName="kubernetes.io/configmap/cf6a6e00-44f7-4af0-be4d-3650c552d729-konnectivity-ca" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935397 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e29718e2-f954-4ec8-9db2-d6c3638bc51c" volumeName="kubernetes.io/projected/e29718e2-f954-4ec8-9db2-d6c3638bc51c-kube-api-access-6mzl9" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.934200 4963 manager.go:319] Starting recovery of all containers Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935829 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24a97420-efcd-4a6f-b072-1e5792264188" volumeName="kubernetes.io/configmap/24a97420-efcd-4a6f-b072-1e5792264188-telemetry-config" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935862 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf6a6e00-44f7-4af0-be4d-3650c552d729" volumeName="kubernetes.io/secret/cf6a6e00-44f7-4af0-be4d-3650c552d729-agent-certs" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935878 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3c555a1-e6bd-4042-be58-404cb6693922" volumeName="kubernetes.io/configmap/e3c555a1-e6bd-4042-be58-404cb6693922-trusted-ca" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935929 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb7b7d50-f5bf-45b4-acaa-f1bc19831287" volumeName="kubernetes.io/projected/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-kube-api-access-5m4g8" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935953 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed5e1db5-564c-4abd-87c4-a2dda264a926" volumeName="kubernetes.io/projected/ed5e1db5-564c-4abd-87c4-a2dda264a926-kube-api-access-fbvs7" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.935968 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f33677f-ee87-4cd1-bdb0-b74000834db2" volumeName="kubernetes.io/projected/1f33677f-ee87-4cd1-bdb0-b74000834db2-kube-api-access-wq6p2" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.936004 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24a97420-efcd-4a6f-b072-1e5792264188" volumeName="kubernetes.io/projected/24a97420-efcd-4a6f-b072-1e5792264188-kube-api-access-8zvs2" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.936032 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2955ecad-cad6-48d2-a2f0-1c1aae3d47f4" volumeName="kubernetes.io/projected/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-api-access-8l7br" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.936048 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="510fbc36-62de-4096-b8e8-d8606a4ad7d5" volumeName="kubernetes.io/configmap/510fbc36-62de-4096-b8e8-d8606a4ad7d5-config" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.936058 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d21a1ae-f712-4f89-b34f-8016fd918484" volumeName="kubernetes.io/empty-dir/8d21a1ae-f712-4f89-b34f-8016fd918484-snapshots" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.936074 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af51f192-4170-4d3b-85ae-7d2ceb14de05" volumeName="kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.936093 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7559e5a-a8fd-450c-8f40-f2072e93a9b5" volumeName="kubernetes.io/empty-dir/c7559e5a-a8fd-450c-8f40-f2072e93a9b5-tmp-dir" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.936104 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff7262fd-b29d-4c72-b717-ef9742bdd2db" volumeName="kubernetes.io/projected/ff7262fd-b29d-4c72-b717-ef9742bdd2db-kube-api-access-8nwhb" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.936120 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12ce6198-3748-49b2-ad3d-f5c3d701fc96" volumeName="kubernetes.io/secret/12ce6198-3748-49b2-ad3d-f5c3d701fc96-signing-key" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.936131 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f33677f-ee87-4cd1-bdb0-b74000834db2" volumeName="kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-trusted-ca-bundle" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.936149 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42d01b00-5ec9-4f8d-a8f2-10e0addd1b67" volumeName="kubernetes.io/secret/42d01b00-5ec9-4f8d-a8f2-10e0addd1b67-cert" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.936164 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a6bf982-7a96-41ae-b24d-78cdabbb7c37" volumeName="kubernetes.io/configmap/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-config" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.936175 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d21a1ae-f712-4f89-b34f-8016fd918484" volumeName="kubernetes.io/empty-dir/8d21a1ae-f712-4f89-b34f-8016fd918484-tmp" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.936187 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2af25ca-9ad2-481c-addd-ebe3d999fc4e" volumeName="kubernetes.io/projected/d2af25ca-9ad2-481c-addd-ebe3d999fc4e-kube-api-access-fhvdt" seLinuxMountContext="" Nov 03 20:47:30.936267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.936201 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eafb0c22-9550-4b1c-8134-fc80562025eb" volumeName="kubernetes.io/projected/eafb0c22-9550-4b1c-8134-fc80562025eb-kube-api-access-2mzjv" seLinuxMountContext="" Nov 03 20:47:30.939485 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.937547 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb02f36e-58c6-47fb-b6f7-0ba78d029ffb" volumeName="kubernetes.io/configmap/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 20:47:30.939485 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.937577 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="510fbc36-62de-4096-b8e8-d8606a4ad7d5" volumeName="kubernetes.io/projected/510fbc36-62de-4096-b8e8-d8606a4ad7d5-kube-api-access-fhh48" seLinuxMountContext="" Nov 03 20:47:30.939485 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.937592 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ea4f6f2-4699-46d7-91bc-8391dde191eb" volumeName="kubernetes.io/projected/5ea4f6f2-4699-46d7-91bc-8391dde191eb-kube-api-access-gkv6p" seLinuxMountContext="" Nov 03 20:47:30.939485 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.937603 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="793c02dd-468b-4308-bac2-67d04e40579c" volumeName="kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-registry-certificates" seLinuxMountContext="" Nov 03 20:47:30.939485 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.937615 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="793c02dd-468b-4308-bac2-67d04e40579c" volumeName="kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-kube-api-access-5cjnm" seLinuxMountContext="" Nov 03 20:47:30.939485 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.937629 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3c555a1-e6bd-4042-be58-404cb6693922" volumeName="kubernetes.io/projected/e3c555a1-e6bd-4042-be58-404cb6693922-kube-api-access-ltgln" seLinuxMountContext="" Nov 03 20:47:30.939485 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.937641 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e863782c-c289-4fa5-93b1-b475358a5001" volumeName="kubernetes.io/projected/e863782c-c289-4fa5-93b1-b475358a5001-kube-api-access-l9dm2" seLinuxMountContext="" Nov 03 20:47:30.939485 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.937654 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34ba944d-b672-489d-9acc-fde356202fbe" volumeName="kubernetes.io/projected/34ba944d-b672-489d-9acc-fde356202fbe-kube-api-access-6zwts" seLinuxMountContext="" Nov 03 20:47:30.939485 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.937668 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="462e871a-1226-4f07-b34c-ef612fa6bf52" volumeName="kubernetes.io/configmap/462e871a-1226-4f07-b34c-ef612fa6bf52-serviceca" seLinuxMountContext="" Nov 03 20:47:30.939485 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.937680 4963 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff7262fd-b29d-4c72-b717-ef9742bdd2db" volumeName="kubernetes.io/empty-dir/ff7262fd-b29d-4c72-b717-ef9742bdd2db-tmp-dir" seLinuxMountContext="" Nov 03 20:47:30.939485 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.937692 4963 reconstruct.go:97] "Volume reconstruction finished" Nov 03 20:47:30.939485 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.937702 4963 reconciler.go:29] "Reconciler: start to sync state" Nov 03 20:47:30.941191 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.941111 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/1.log" Nov 03 20:47:30.943611 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.943557 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/1.log" Nov 03 20:47:30.958093 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.958060 4963 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 20:47:30.959292 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.959273 4963 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 20:47:30.959292 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.959295 4963 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 20:47:30.959472 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.959314 4963 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 20:47:30.959472 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:30.959348 4963 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 20:47:30.978396 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.978342 4963 generic.go:356] "Generic (PLEG): container finished" podID="6c672b05e20cfe6494452bceff1f2bac" containerID="aafc8793f8121d4fb45fd3599a567b38ee778780783a6de3055486890eab708b" exitCode=0 Nov 03 20:47:30.988143 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.988115 4963 generic.go:356] "Generic (PLEG): container finished" podID="fb02f36e-58c6-47fb-b6f7-0ba78d029ffb" containerID="30a04279adaae404f1d16cc19a01b2420f311187ddfadc4e43852d6f83c312f8" exitCode=0 Nov 03 20:47:30.988143 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.988138 4963 generic.go:356] "Generic (PLEG): container finished" podID="fb02f36e-58c6-47fb-b6f7-0ba78d029ffb" containerID="ae6a52571df0bd4f4777619ad703b9db45e3162a9cd9f36e960f6916cf71c0d4" exitCode=0 Nov 03 20:47:30.988143 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.988146 4963 generic.go:356] "Generic (PLEG): container finished" podID="fb02f36e-58c6-47fb-b6f7-0ba78d029ffb" containerID="31673d96c7942ccf8c163b6db8df1a65fcf6d62c34f82359766527b25d308257" exitCode=0 Nov 03 20:47:30.988378 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.988153 4963 generic.go:356] "Generic (PLEG): container finished" podID="fb02f36e-58c6-47fb-b6f7-0ba78d029ffb" containerID="d064963297fba07484bd3b9d45b5d0f2580f6b14892fdec69bb91cbb35989123" exitCode=0 Nov 03 20:47:30.988378 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.988159 4963 generic.go:356] "Generic (PLEG): container finished" podID="fb02f36e-58c6-47fb-b6f7-0ba78d029ffb" containerID="564fbee1f50e6deca900e61bc9758d4dad87f126e9bdf969802345680218971d" exitCode=0 Nov 03 20:47:30.988378 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:30.988165 4963 generic.go:356] "Generic (PLEG): container finished" podID="fb02f36e-58c6-47fb-b6f7-0ba78d029ffb" containerID="6c525f10a9c77c8cb2fdd2be3dc983e4d260820427f09323cd79622ce92d1af0" exitCode=0 Nov 03 20:47:31.013077 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.013051 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/1.log" Nov 03 20:47:31.013239 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.013083 4963 generic.go:356] "Generic (PLEG): container finished" podID="e3c555a1-e6bd-4042-be58-404cb6693922" containerID="29cf45ae230e23b9bed36cf8ecbd1b8c7222aa0b362a7fee1f51cdfc7b569516" exitCode=255 Nov 03 20:47:31.028743 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:31.028716 4963 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-134.us-west-2.compute.internal\" not found" Nov 03 20:47:31.033502 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.033482 4963 manager.go:324] Recovery completed Nov 03 20:47:31.053762 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.053735 4963 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:47:31.055019 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.054997 4963 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:47:31.055117 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.055035 4963 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:47:31.055117 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.055046 4963 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:47:31.057149 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.057132 4963 cpu_manager.go:226] "Starting" policy="none" Nov 03 20:47:31.057149 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.057147 4963 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 20:47:31.057241 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.057151 4963 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:47:31.057241 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.057166 4963 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 20:47:31.057312 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.057299 4963 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 20:47:31.057348 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.057312 4963 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 20:47:31.057348 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.057332 4963 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 20:47:31.057348 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.057337 4963 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 20:47:31.057348 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.057342 4963 policy_none.go:50] "Start" Nov 03 20:47:31.057462 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.057351 4963 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 20:47:31.057462 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.057360 4963 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 20:47:31.057513 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.057493 4963 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 20:47:31.057513 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.057500 4963 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 20:47:31.057513 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.057506 4963 policy_none.go:44] "Start" Nov 03 20:47:31.057978 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.057962 4963 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:47:31.058063 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.057992 4963 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:47:31.058063 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.058007 4963 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:47:31.059414 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:31.059399 4963 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 20:47:31.062577 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.062546 4963 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 20:47:31.065296 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.065280 4963 manager.go:341] "Starting Device Plugin manager" Nov 03 20:47:31.065390 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:31.065307 4963 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 20:47:31.065390 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.065317 4963 server.go:87] "Starting device plugin registration server" Nov 03 20:47:31.065609 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.065598 4963 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 20:47:31.065655 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.065610 4963 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 20:47:31.065706 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.065677 4963 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 20:47:31.065851 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.065815 4963 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 20:47:31.065960 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.065855 4963 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 20:47:31.065960 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.065862 4963 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 20:47:31.066071 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.065958 4963 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:47:31.065845751Z","UUID":"6af02660-04a1-40f2-9487-ad5cae1f9bfa","Handler":null,"Name":"","Endpoint":""} Nov 03 20:47:31.067186 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.067144 4963 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:47:31.067566 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:31.067542 4963 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 20:47:31.067626 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:31.067581 4963 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-134.us-west-2.compute.internal\" not found" Nov 03 20:47:31.067626 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.067545 4963 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 20:47:31.067626 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.067610 4963 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 20:47:31.110127 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.110090 4963 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:47:31.161159 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.161123 4963 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 20:47:31.161329 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.161278 4963 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:47:31.162215 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.162197 4963 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:47:31.166215 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.166196 4963 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:47:31.167252 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.167235 4963 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:47:31.167333 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.167266 4963 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:47:31.167333 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.167282 4963 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:47:31.167405 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.167337 4963 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:31.259777 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.259686 4963 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal"] Nov 03 20:47:31.260357 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.260323 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" event={"ID":"6c672b05e20cfe6494452bceff1f2bac","Type":"ContainerStarted","Data":"99787d0dd813df80f11e4ae56e40c10cf9f49cddaf2f0505b2770d385dfeefe1"} Nov 03 20:47:31.260477 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.260360 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" event={"ID":"6c672b05e20cfe6494452bceff1f2bac","Type":"ContainerDied","Data":"aafc8793f8121d4fb45fd3599a567b38ee778780783a6de3055486890eab708b"} Nov 03 20:47:31.260477 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.260376 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" event={"ID":"6c672b05e20cfe6494452bceff1f2bac","Type":"ContainerStarted","Data":"7069e383c5e90e1f48fc0dce4f850b630e915add3755f25704bc7093cbdeae28"} Nov 03 20:47:31.260477 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.260439 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal" event={"ID":"954da78189a5577b346472d086419753","Type":"ContainerStarted","Data":"4d64fa7e8a094152280991f1982732bd4b514c71894808c8488261dbfbc60947"} Nov 03 20:47:31.260477 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.260450 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal" event={"ID":"954da78189a5577b346472d086419753","Type":"ContainerStarted","Data":"f2cf2855e1e18cbff8c7c4f4ff8c2b8b63020f9b606a416adfd2f86609337bb7"} Nov 03 20:47:31.260638 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.260583 4963 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:31.260638 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.260604 4963 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:31.288691 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.288658 4963 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:31.288816 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.288733 4963 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:31.321055 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.321017 4963 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:47:31.321202 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:31.321166 4963 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:31.321432 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.321411 4963 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:47:31.321488 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:31.321472 4963 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:31.362097 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.362065 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c672b05e20cfe6494452bceff1f2bac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal\" (UID: \"6c672b05e20cfe6494452bceff1f2bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:31.362283 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.362103 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c672b05e20cfe6494452bceff1f2bac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal\" (UID: \"6c672b05e20cfe6494452bceff1f2bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:31.362283 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.362128 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/954da78189a5577b346472d086419753-config\") pod \"kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal\" (UID: \"954da78189a5577b346472d086419753\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:31.462489 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.462457 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c672b05e20cfe6494452bceff1f2bac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal\" (UID: \"6c672b05e20cfe6494452bceff1f2bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:31.462489 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.462492 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/954da78189a5577b346472d086419753-config\") pod \"kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal\" (UID: \"954da78189a5577b346472d086419753\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:31.462742 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.462509 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c672b05e20cfe6494452bceff1f2bac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal\" (UID: \"6c672b05e20cfe6494452bceff1f2bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:31.462742 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.462560 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/954da78189a5577b346472d086419753-config\") pod \"kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal\" (UID: \"954da78189a5577b346472d086419753\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:31.462742 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.462573 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c672b05e20cfe6494452bceff1f2bac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal\" (UID: \"6c672b05e20cfe6494452bceff1f2bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:31.462742 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.462606 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c672b05e20cfe6494452bceff1f2bac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal\" (UID: \"6c672b05e20cfe6494452bceff1f2bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:31.905001 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:31.904965 4963 apiserver.go:52] "Watching apiserver" Nov 03 20:47:32.020478 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.020453 4963 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:32.020648 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.020633 4963 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:32.051235 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.051202 4963 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:47:32.052007 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.051977 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9c99h","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4","openshift-multus/multus-snszh","openshift-multus/network-metrics-daemon-mr55c","kube-system/kas-connection-checker-685895c7c9-bd5l5","openshift-insights/insights-runtime-extractor-d8xh9","openshift-ovn-kubernetes/ovnkube-node-vkh7b","openshift-multus/multus-additional-cni-plugins-tgl9l","kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal","openshift-image-registry/node-ca-r9lld","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal","openshift-service-ca/service-ca-5fffc97bdf-vcjdr","kube-system/global-pull-secret-syncer-lvbhk","openshift-dns/dns-default-njb7z","openshift-dns/node-resolver-sm2mn","openshift-image-registry/image-registry-75999699c7-pfc8g","kube-system/kas-connection-checker-685895c7c9-ghxn6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn","openshift-ingress/router-default-7fb544c95-8798s","openshift-network-diagnostics/network-check-source-6f78d59f99-t4tng","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp","openshift-insights/periodic-gathering-4dvbp-dpsms","openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc","openshift-network-operator/iptables-alerter-wl57x","openshift-network-diagnostics/network-check-target-l7lxn","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nm757","openshift-image-registry/image-registry-687954fc7b-25pdn","openshift-network-console/networking-console-plugin-86c478bb88-xwt2z","openshift-kube-storage-version-migrator/migrator-66df5d496b-ndsnt","openshift-kube-storage-version-migrator/migrator-66df5d496b-qlrbs","openshift-cluster-node-tuning-operator/tuned-7m7vq","openshift-insights/insights-operator-7c4986d449-92hts","openshift-console-operator/console-operator-7cd56b8dbc-ccxnp","openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk","kube-system/konnectivity-agent-j5mgz","kube-system/kas-connection-checker-685895c7c9-kwzxh"] Nov 03 20:47:32.053490 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.053468 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:32.054451 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.054437 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:32.055488 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.055464 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:32.055927 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.055896 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:32.059731 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.059704 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:32.065088 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.065071 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:32.066087 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.066072 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:32.067100 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.067086 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:32.068142 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.068130 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:32.070233 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.070213 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" Nov 03 20:47:32.078462 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.078445 4963 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:47:32.078556 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.078493 4963 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:32.078924 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.078897 4963 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:47:32.078966 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.078944 4963 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.us-west-2.compute.internal" Nov 03 20:47:32.103586 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.103550 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 20:47:32.103586 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.103549 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 20:47:32.103778 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.103748 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-92gvj\"" Nov 03 20:47:32.103819 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.103794 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-p57f6\"" Nov 03 20:47:32.103854 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.103836 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-d5cfx\"" Nov 03 20:47:32.104082 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.104069 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 20:47:32.104082 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.104076 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.104154 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.104078 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.104319 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.104304 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.104439 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.104425 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.104611 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.104596 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.104779 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.104765 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 20:47:32.105109 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105093 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 20:47:32.105164 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105126 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j76n8\"" Nov 03 20:47:32.105305 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105291 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 20:47:32.105381 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105366 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 20:47:32.105466 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105442 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 20:47:32.105549 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105525 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.105549 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105525 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 20:47:32.105648 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105572 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 20:47:32.105648 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105625 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.105743 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105662 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 20:47:32.105743 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105679 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.105743 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105680 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.105743 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105701 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.105926 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105779 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:47:32.105926 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105798 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.105926 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105799 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.105926 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105834 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.105926 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105856 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-g5tq4\"" Nov 03 20:47:32.105926 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.105878 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 20:47:32.106142 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.106069 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.106255 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.106240 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 20:47:32.106317 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.106296 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lw8vn\"" Nov 03 20:47:32.106464 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.106452 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-99tc2\"" Nov 03 20:47:32.130920 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.130878 4963 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 20:47:32.155937 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.155836 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 20:47:32.155937 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.155895 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 20:47:32.156136 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156000 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.156136 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156027 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:47:32.156136 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156035 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.156136 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156055 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vwxfp\"" Nov 03 20:47:32.156136 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156125 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.156380 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156153 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 20:47:32.156380 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156174 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:47:32.156380 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156217 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.156380 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156241 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.156380 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156267 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.156380 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156285 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7vv6z\"" Nov 03 20:47:32.156380 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156352 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.156380 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156373 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 20:47:32.156704 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156422 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-s2xmz\"" Nov 03 20:47:32.156704 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156432 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-78pt4\"" Nov 03 20:47:32.156704 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156472 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.156704 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156506 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 20:47:32.156704 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156538 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 20:47:32.156704 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156544 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 20:47:32.156704 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156565 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.156704 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156577 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.156704 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156582 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.156704 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156637 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.156704 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156663 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:47:32.156704 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156690 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 20:47:32.157214 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156714 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.157214 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156723 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 20:47:32.157214 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156835 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.157214 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156870 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tlf9q\"" Nov 03 20:47:32.157214 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156923 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-p5wrg\"" Nov 03 20:47:32.157214 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156930 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-868xx\"" Nov 03 20:47:32.157214 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.156717 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 20:47:32.157214 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.157018 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xldt8\"" Nov 03 20:47:32.157214 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.157029 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:47:32.157214 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.157045 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 20:47:32.157214 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.157024 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.157214 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.157047 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:47:32.157214 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.157087 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:47:32.157214 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.157089 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.157214 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.157101 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.157745 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.157341 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rf5zr\"" Nov 03 20:47:32.157745 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.157389 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gf6gt\"" Nov 03 20:47:32.157745 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.157427 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-964vd\"" Nov 03 20:47:32.157745 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.157472 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ldspv\"" Nov 03 20:47:32.157745 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.157496 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:47:32.157939 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.157836 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:47:32.157939 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.157929 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 20:47:32.158049 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.157932 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:47:32.166065 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166038 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb96b430-8876-41db-a35c-19db3cee7635-nginx-conf\") pod \"networking-console-plugin-86c478bb88-xwt2z\" (UID: \"fb96b430-8876-41db-a35c-19db3cee7635\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:32.166141 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166071 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1f33677f-ee87-4cd1-bdb0-b74000834db2-archives-path\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:32.166141 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166091 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-run\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.166141 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166112 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvs2\" (UniqueName: \"kubernetes.io/projected/24a97420-efcd-4a6f-b072-1e5792264188-kube-api-access-8zvs2\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:32.166141 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166130 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-image-registry-private-configuration\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:32.166269 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166226 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1f33677f-ee87-4cd1-bdb0-b74000834db2-archives-path\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:32.166269 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166230 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-trusted-ca\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:32.166356 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166296 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:32.166356 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166333 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhh48\" (UniqueName: \"kubernetes.io/projected/510fbc36-62de-4096-b8e8-d8606a4ad7d5-kube-api-access-fhh48\") pod \"service-ca-operator-544d749c8c-s82vc\" (UID: \"510fbc36-62de-4096-b8e8-d8606a4ad7d5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" Nov 03 20:47:32.166450 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166360 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brl7b\" (UniqueName: \"kubernetes.io/projected/d0d6aa7f-3a2b-482f-8339-291a11a77c22-kube-api-access-brl7b\") pod \"network-check-source-6f78d59f99-t4tng\" (UID: \"d0d6aa7f-3a2b-482f-8339-291a11a77c22\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4tng" Nov 03 20:47:32.166450 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166424 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/12ce6198-3748-49b2-ad3d-f5c3d701fc96-signing-cabundle\") pod \"service-ca-5fffc97bdf-vcjdr\" (UID: \"12ce6198-3748-49b2-ad3d-f5c3d701fc96\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" Nov 03 20:47:32.166450 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166433 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb96b430-8876-41db-a35c-19db3cee7635-nginx-conf\") pod \"networking-console-plugin-86c478bb88-xwt2z\" (UID: \"fb96b430-8876-41db-a35c-19db3cee7635\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:32.166590 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166459 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e456784-7491-426a-80e5-17a9dfa2f07f-ovn-node-metrics-cert\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.166590 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166488 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-var-lib-kubelet\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.166590 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166499 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-image-registry-private-configuration\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:32.166590 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166513 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.166590 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166564 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-cnibin\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:47:32.166801 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166605 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbvs7\" (UniqueName: \"kubernetes.io/projected/ed5e1db5-564c-4abd-87c4-a2dda264a926-kube-api-access-fbvs7\") pod \"migrator-66df5d496b-qlrbs\" (UID: \"ed5e1db5-564c-4abd-87c4-a2dda264a926\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlrbs" Nov 03 20:47:32.166801 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166641 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e456784-7491-426a-80e5-17a9dfa2f07f-ovn-node-metrics-cert\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.166801 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166646 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d21a1ae-f712-4f89-b34f-8016fd918484-tmp\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:32.166801 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166674 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-slash\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.166801 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166697 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-cni-netd\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.166801 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166721 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-registry-certificates\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:32.166801 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166723 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-trusted-ca\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:32.166801 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166749 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d21a1ae-f712-4f89-b34f-8016fd918484-tmp\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:32.167080 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166863 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-registry-certificates\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:32.167080 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166865 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-host\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.167080 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166893 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mzjv\" (UniqueName: \"kubernetes.io/projected/eafb0c22-9550-4b1c-8134-fc80562025eb-kube-api-access-2mzjv\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.167080 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166931 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-var-lib-openvswitch\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.167080 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166949 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-crio-socket\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:32.167080 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166964 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-cni-binary-copy\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:47:32.167080 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166978 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-run-ovn\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.167080 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.166993 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-conf-dir\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.167080 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167027 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-etc-kubernetes\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.167414 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167138 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxhlc\" (UniqueName: \"kubernetes.io/projected/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-kube-api-access-gxhlc\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.167414 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167177 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:32.167414 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167228 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cf6a6e00-44f7-4af0-be4d-3650c552d729-agent-certs\") pod \"konnectivity-agent-j5mgz\" (UID: \"cf6a6e00-44f7-4af0-be4d-3650c552d729\") " pod="kube-system/konnectivity-agent-j5mgz" Nov 03 20:47:32.167414 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167278 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbfzh\" (UniqueName: \"kubernetes.io/projected/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-kube-api-access-dbfzh\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:47:32.167414 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167313 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-cni-binary-copy\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:47:32.167414 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167390 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.167414 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167412 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-cnibin\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.167648 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167429 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-run-multus-certs\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.167648 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167448 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m4g8\" (UniqueName: \"kubernetes.io/projected/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-kube-api-access-5m4g8\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:32.167648 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167464 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvp8m\" (UniqueName: \"kubernetes.io/projected/542dd56a-cfab-40e2-a01b-a172651d46f8-kube-api-access-cvp8m\") pod \"iptables-alerter-wl57x\" (UID: \"542dd56a-cfab-40e2-a01b-a172651d46f8\") " pod="openshift-network-operator/iptables-alerter-wl57x" Nov 03 20:47:32.167648 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167498 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-etc-openvswitch\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.167648 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167554 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cf6a6e00-44f7-4af0-be4d-3650c552d729-agent-certs\") pod \"konnectivity-agent-j5mgz\" (UID: \"cf6a6e00-44f7-4af0-be4d-3650c552d729\") " pod="kube-system/konnectivity-agent-j5mgz" Nov 03 20:47:32.167648 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167571 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-image-registry-private-configuration\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:32.167838 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167697 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:32.167838 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167732 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-run-systemd\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.167838 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167744 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-image-registry-private-configuration\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:32.167838 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167764 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t42hf\" (UniqueName: \"kubernetes.io/projected/c7559e5a-a8fd-450c-8f40-f2072e93a9b5-kube-api-access-t42hf\") pod \"node-resolver-sm2mn\" (UID: \"c7559e5a-a8fd-450c-8f40-f2072e93a9b5\") " pod="openshift-dns/node-resolver-sm2mn" Nov 03 20:47:32.167838 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167792 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3c555a1-e6bd-4042-be58-404cb6693922-serving-cert\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:32.167838 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167811 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-service-ca-bundle\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:32.168135 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167925 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:32.168135 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167945 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-service-ca-bundle\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:32.168135 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167952 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3c555a1-e6bd-4042-be58-404cb6693922-serving-cert\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:32.168135 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167953 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-tuned\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.168135 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.167988 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bxrc\" (UniqueName: \"kubernetes.io/projected/12ce6198-3748-49b2-ad3d-f5c3d701fc96-kube-api-access-6bxrc\") pod \"service-ca-5fffc97bdf-vcjdr\" (UID: \"12ce6198-3748-49b2-ad3d-f5c3d701fc96\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" Nov 03 20:47:32.168135 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168014 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d21a1ae-f712-4f89-b34f-8016fd918484-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:32.168135 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168019 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-tuned\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.168135 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168041 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hd42\" (UniqueName: \"kubernetes.io/projected/8d21a1ae-f712-4f89-b34f-8016fd918484-kube-api-access-2hd42\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:32.168135 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168073 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7n4p\" (UniqueName: \"kubernetes.io/projected/8e456784-7491-426a-80e5-17a9dfa2f07f-kube-api-access-v7n4p\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.168135 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168093 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq6p2\" (UniqueName: \"kubernetes.io/projected/1f33677f-ee87-4cd1-bdb0-b74000834db2-kube-api-access-wq6p2\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:32.168135 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168111 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:32.168135 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168133 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-node-log\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.168695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168187 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zwts\" (UniqueName: \"kubernetes.io/projected/34ba944d-b672-489d-9acc-fde356202fbe-kube-api-access-6zwts\") pod \"kas-connection-checker-685895c7c9-ghxn6\" (UID: \"34ba944d-b672-489d-9acc-fde356202fbe\") " pod="kube-system/kas-connection-checker-685895c7c9-ghxn6" Nov 03 20:47:32.168695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168208 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6rc\" (UniqueName: \"kubernetes.io/projected/462e871a-1226-4f07-b34c-ef612fa6bf52-kube-api-access-pc6rc\") pod \"node-ca-r9lld\" (UID: \"462e871a-1226-4f07-b34c-ef612fa6bf52\") " pod="openshift-image-registry/node-ca-r9lld" Nov 03 20:47:32.168695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168243 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-trusted-ca-bundle\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:32.168695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168260 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:32.168695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168284 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-systemd\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.168695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168299 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-sysconfig\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.168695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168308 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d21a1ae-f712-4f89-b34f-8016fd918484-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:32.168695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168314 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-registration-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.168695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168377 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkv6p\" (UniqueName: \"kubernetes.io/projected/5ea4f6f2-4699-46d7-91bc-8391dde191eb-kube-api-access-gkv6p\") pod \"kas-connection-checker-685895c7c9-kwzxh\" (UID: \"5ea4f6f2-4699-46d7-91bc-8391dde191eb\") " pod="kube-system/kas-connection-checker-685895c7c9-kwzxh" Nov 03 20:47:32.168695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168407 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cf6a6e00-44f7-4af0-be4d-3650c552d729-konnectivity-ca\") pod \"konnectivity-agent-j5mgz\" (UID: \"cf6a6e00-44f7-4af0-be4d-3650c552d729\") " pod="kube-system/konnectivity-agent-j5mgz" Nov 03 20:47:32.168695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168433 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e29718e2-f954-4ec8-9db2-d6c3638bc51c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4\" (UID: \"e29718e2-f954-4ec8-9db2-d6c3638bc51c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" Nov 03 20:47:32.168695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168462 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-run-netns\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.168695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168486 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-kubernetes\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.168695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168510 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-device-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.168695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168535 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:47:32.168695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168562 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-system-cni-dir\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.168695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168602 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-trusted-ca-bundle\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:32.168695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168707 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3c555a1-e6bd-4042-be58-404cb6693922-config\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168726 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xwt2z\" (UID: \"fb96b430-8876-41db-a35c-19db3cee7635\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168744 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/462e871a-1226-4f07-b34c-ef612fa6bf52-serviceca\") pod \"node-ca-r9lld\" (UID: \"462e871a-1226-4f07-b34c-ef612fa6bf52\") " pod="openshift-image-registry/node-ca-r9lld" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168745 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cf6a6e00-44f7-4af0-be4d-3650c552d729-konnectivity-ca\") pod \"konnectivity-agent-j5mgz\" (UID: \"cf6a6e00-44f7-4af0-be4d-3650c552d729\") " pod="kube-system/konnectivity-agent-j5mgz" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168758 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-sys\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168763 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e29718e2-f954-4ec8-9db2-d6c3638bc51c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4\" (UID: \"e29718e2-f954-4ec8-9db2-d6c3638bc51c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168776 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9dm2\" (UniqueName: \"kubernetes.io/projected/e863782c-c289-4fa5-93b1-b475358a5001-kube-api-access-l9dm2\") pod \"kas-connection-checker-685895c7c9-bd5l5\" (UID: \"e863782c-c289-4fa5-93b1-b475358a5001\") " pod="kube-system/kas-connection-checker-685895c7c9-bd5l5" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168819 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168989 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/462e871a-1226-4f07-b34c-ef612fa6bf52-serviceca\") pod \"node-ca-r9lld\" (UID: \"462e871a-1226-4f07-b34c-ef612fa6bf52\") " pod="openshift-image-registry/node-ca-r9lld" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.168996 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-socket-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169021 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-cni-bin\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169030 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3c555a1-e6bd-4042-be58-404cb6693922-config\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169057 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-installation-pull-secrets\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169079 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nwhb\" (UniqueName: \"kubernetes.io/projected/ff7262fd-b29d-4c72-b717-ef9742bdd2db-kube-api-access-8nwhb\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169145 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-sys-fs\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169243 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mzl9\" (UniqueName: \"kubernetes.io/projected/e29718e2-f954-4ec8-9db2-d6c3638bc51c-kube-api-access-6mzl9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4\" (UID: \"e29718e2-f954-4ec8-9db2-d6c3638bc51c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169281 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169314 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-installation-pull-secrets\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169381 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-certificates\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169421 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l7br\" (UniqueName: \"kubernetes.io/projected/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-api-access-8l7br\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169446 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d21a1ae-f712-4f89-b34f-8016fd918484-serving-cert\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:32.169561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169508 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-var-lib-cni-bin\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169579 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-certificates\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169595 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/462e871a-1226-4f07-b34c-ef612fa6bf52-host\") pod \"node-ca-r9lld\" (UID: \"462e871a-1226-4f07-b34c-ef612fa6bf52\") " pod="openshift-image-registry/node-ca-r9lld" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169614 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d21a1ae-f712-4f89-b34f-8016fd918484-serving-cert\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169633 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-data-volume\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169654 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-run-openvswitch\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169697 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/793c02dd-468b-4308-bac2-67d04e40579c-ca-trust-extracted\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169706 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-data-volume\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169754 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/793c02dd-468b-4308-bac2-67d04e40579c-ca-trust-extracted\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169784 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cjnm\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-kube-api-access-5cjnm\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169823 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-run-k8s-cni-cncf-io\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169914 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-kubelet-config\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169942 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-bound-sa-token\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169966 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/12ce6198-3748-49b2-ad3d-f5c3d701fc96-signing-key\") pod \"service-ca-5fffc97bdf-vcjdr\" (UID: \"12ce6198-3748-49b2-ad3d-f5c3d701fc96\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.169984 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170007 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e29718e2-f954-4ec8-9db2-d6c3638bc51c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4\" (UID: \"e29718e2-f954-4ec8-9db2-d6c3638bc51c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170067 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-daemon-config\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170149 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-dbus\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170193 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e29718e2-f954-4ec8-9db2-d6c3638bc51c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4\" (UID: \"e29718e2-f954-4ec8-9db2-d6c3638bc51c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170243 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/510fbc36-62de-4096-b8e8-d8606a4ad7d5-config\") pod \"service-ca-operator-544d749c8c-s82vc\" (UID: \"510fbc36-62de-4096-b8e8-d8606a4ad7d5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" Nov 03 20:47:32.170274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170272 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-system-cni-dir\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170326 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-daemon-config\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170326 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e456784-7491-426a-80e5-17a9dfa2f07f-ovnkube-script-lib\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170406 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/510fbc36-62de-4096-b8e8-d8606a4ad7d5-config\") pod \"service-ca-operator-544d749c8c-s82vc\" (UID: \"510fbc36-62de-4096-b8e8-d8606a4ad7d5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170414 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7559e5a-a8fd-450c-8f40-f2072e93a9b5-tmp-dir\") pod \"node-resolver-sm2mn\" (UID: \"c7559e5a-a8fd-450c-8f40-f2072e93a9b5\") " pod="openshift-dns/node-resolver-sm2mn" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170434 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-socket-dir-parent\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170450 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170462 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7559e5a-a8fd-450c-8f40-f2072e93a9b5-tmp-dir\") pod \"node-resolver-sm2mn\" (UID: \"c7559e5a-a8fd-450c-8f40-f2072e93a9b5\") " pod="openshift-dns/node-resolver-sm2mn" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170468 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170505 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/15d37b2c-d051-419e-b0cb-d761cac6ff74-ca-trust-extracted\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170526 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-sysctl-d\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170550 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e456784-7491-426a-80e5-17a9dfa2f07f-ovnkube-script-lib\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170553 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-var-lib-kubelet\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170575 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/15d37b2c-d051-419e-b0cb-d761cac6ff74-ca-trust-extracted\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170586 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3c555a1-e6bd-4042-be58-404cb6693922-trusted-ca\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170709 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-default-certificate\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170742 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-stats-auth\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170773 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c22-9550-4b1c-8134-fc80562025eb-tmp\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170808 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/542dd56a-cfab-40e2-a01b-a172651d46f8-host-slash\") pod \"iptables-alerter-wl57x\" (UID: \"542dd56a-cfab-40e2-a01b-a172651d46f8\") " pod="openshift-network-operator/iptables-alerter-wl57x" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170849 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170868 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3c555a1-e6bd-4042-be58-404cb6693922-trusted-ca\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170877 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170935 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7559e5a-a8fd-450c-8f40-f2072e93a9b5-hosts-file\") pod \"node-resolver-sm2mn\" (UID: \"c7559e5a-a8fd-450c-8f40-f2072e93a9b5\") " pod="openshift-dns/node-resolver-sm2mn" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170943 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c22-9550-4b1c-8134-fc80562025eb-tmp\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170963 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-default-certificate\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.170976 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/24a97420-efcd-4a6f-b072-1e5792264188-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171008 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vqts\" (UniqueName: \"kubernetes.io/projected/84556453-e5d9-4449-ba59-819c248e1e2a-kube-api-access-4vqts\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171020 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-stats-auth\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:32.171058 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171063 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-trusted-ca\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171122 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-systemd-units\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171142 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e456784-7491-426a-80e5-17a9dfa2f07f-env-overrides\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171160 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-run-netns\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171175 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-hostroot\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171179 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/24a97420-efcd-4a6f-b072-1e5792264188-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171224 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltgln\" (UniqueName: \"kubernetes.io/projected/e3c555a1-e6bd-4042-be58-404cb6693922-kube-api-access-ltgln\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171268 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76nsp\" (UniqueName: \"kubernetes.io/projected/245acdd3-cebd-49fe-bf1e-867df120ac8e-kube-api-access-76nsp\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171309 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8d21a1ae-f712-4f89-b34f-8016fd918484-snapshots\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171369 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e456784-7491-426a-80e5-17a9dfa2f07f-env-overrides\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171395 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8d21a1ae-f712-4f89-b34f-8016fd918484-snapshots\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171402 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-trusted-ca\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171415 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nd4m\" (UniqueName: \"kubernetes.io/projected/8f39698b-0204-4492-ac16-e1728e71b8d6-kube-api-access-8nd4m\") pod \"volume-data-source-validator-58c9f5f46d-nm757\" (UID: \"8f39698b-0204-4492-ac16-e1728e71b8d6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nm757" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171461 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd729\" (UniqueName: \"kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729\") pod \"network-check-target-l7lxn\" (UID: \"af51f192-4170-4d3b-85ae-7d2ceb14de05\") " pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171532 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhvdt\" (UniqueName: \"kubernetes.io/projected/d2af25ca-9ad2-481c-addd-ebe3d999fc4e-kube-api-access-fhvdt\") pod \"migrator-66df5d496b-ndsnt\" (UID: \"d2af25ca-9ad2-481c-addd-ebe3d999fc4e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ndsnt" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171609 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-sysctl-conf\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171707 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-lib-modules\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171750 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171785 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-bound-sa-token\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:32.171895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171838 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-cni-dir\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.172460 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171933 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-var-lib-cni-multus\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.172460 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.171969 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj7td\" (UniqueName: \"kubernetes.io/projected/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-kube-api-access-sj7td\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:32.172460 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172026 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42d01b00-5ec9-4f8d-a8f2-10e0addd1b67-cert\") pod \"ingress-canary-9c99h\" (UID: \"42d01b00-5ec9-4f8d-a8f2-10e0addd1b67\") " pod="openshift-ingress-canary/ingress-canary-9c99h" Nov 03 20:47:32.172460 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172055 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/542dd56a-cfab-40e2-a01b-a172651d46f8-iptables-alerter-script\") pod \"iptables-alerter-wl57x\" (UID: \"542dd56a-cfab-40e2-a01b-a172651d46f8\") " pod="openshift-network-operator/iptables-alerter-wl57x" Nov 03 20:47:32.172460 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172082 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e456784-7491-426a-80e5-17a9dfa2f07f-ovnkube-config\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.172460 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172109 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:32.172460 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172142 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdvvx\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-kube-api-access-cdvvx\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:32.172460 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172183 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/510fbc36-62de-4096-b8e8-d8606a4ad7d5-serving-cert\") pod \"service-ca-operator-544d749c8c-s82vc\" (UID: \"510fbc36-62de-4096-b8e8-d8606a4ad7d5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" Nov 03 20:47:32.172460 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172329 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e456784-7491-426a-80e5-17a9dfa2f07f-ovnkube-config\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.172460 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172337 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-log-socket\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.172460 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172347 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/542dd56a-cfab-40e2-a01b-a172651d46f8-iptables-alerter-script\") pod \"iptables-alerter-wl57x\" (UID: \"542dd56a-cfab-40e2-a01b-a172651d46f8\") " pod="openshift-network-operator/iptables-alerter-wl57x" Nov 03 20:47:32.172460 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172380 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-cni-binary-copy\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.172460 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172409 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-installation-pull-secrets\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:32.172460 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172429 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/510fbc36-62de-4096-b8e8-d8606a4ad7d5-serving-cert\") pod \"service-ca-operator-544d749c8c-s82vc\" (UID: \"510fbc36-62de-4096-b8e8-d8606a4ad7d5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" Nov 03 20:47:32.172460 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172436 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fflvq\" (UniqueName: \"kubernetes.io/projected/42d01b00-5ec9-4f8d-a8f2-10e0addd1b67-kube-api-access-fflvq\") pod \"ingress-canary-9c99h\" (UID: \"42d01b00-5ec9-4f8d-a8f2-10e0addd1b67\") " pod="openshift-ingress-canary/ingress-canary-9c99h" Nov 03 20:47:32.172856 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172536 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-cni-binary-copy\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.172856 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172564 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:47:32.172856 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172604 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:32.172856 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172609 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-installation-pull-secrets\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:32.172856 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172626 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff7262fd-b29d-4c72-b717-ef9742bdd2db-tmp-dir\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:32.172856 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172654 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-modprobe-d\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.172856 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172679 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff7262fd-b29d-4c72-b717-ef9742bdd2db-tmp-dir\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:32.172856 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172719 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d21a1ae-f712-4f89-b34f-8016fd918484-service-ca-bundle\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:32.172856 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172820 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d21a1ae-f712-4f89-b34f-8016fd918484-service-ca-bundle\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:32.172856 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172828 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7262fd-b29d-4c72-b717-ef9742bdd2db-config-volume\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:32.172856 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.172858 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-kubelet\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.203782 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.203748 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.203977 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.203955 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.204167 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204140 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:47:32.204167 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204154 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:47:32.204167 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204148 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:47:32.204409 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204225 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:47:32.204409 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204237 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.204409 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204247 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.204409 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204254 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 20:47:32.204409 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204249 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.204409 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204238 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.204409 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204294 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:47:32.204409 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204311 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-68sfv\"" Nov 03 20:47:32.204409 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204321 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 20:47:32.204409 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204312 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mxhpl\"" Nov 03 20:47:32.204409 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204379 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.204409 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204405 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:47:32.204409 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204412 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:47:32.204804 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204447 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:47:32.204804 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204531 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d7jn9\"" Nov 03 20:47:32.204804 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204540 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:47:32.204804 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204582 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l9gsz\"" Nov 03 20:47:32.204804 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204603 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:47:32.204804 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204608 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2g8j2\"" Nov 03 20:47:32.204804 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204653 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-snm99\"" Nov 03 20:47:32.204804 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204659 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:47:32.204804 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204689 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-54hgz\"" Nov 03 20:47:32.204804 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204743 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bqd6h\"" Nov 03 20:47:32.204804 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.204770 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r6str\"" Nov 03 20:47:32.207508 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.207492 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/12ce6198-3748-49b2-ad3d-f5c3d701fc96-signing-cabundle\") pod \"service-ca-5fffc97bdf-vcjdr\" (UID: \"12ce6198-3748-49b2-ad3d-f5c3d701fc96\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" Nov 03 20:47:32.208560 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.208545 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-original-pull-secret\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:32.209293 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.209276 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:47:32.209497 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.209474 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:32.210430 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.210414 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/12ce6198-3748-49b2-ad3d-f5c3d701fc96-signing-key\") pod \"service-ca-5fffc97bdf-vcjdr\" (UID: \"12ce6198-3748-49b2-ad3d-f5c3d701fc96\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" Nov 03 20:47:32.211552 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.211535 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:47:32.212698 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.212675 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42d01b00-5ec9-4f8d-a8f2-10e0addd1b67-cert\") pod \"ingress-canary-9c99h\" (UID: \"42d01b00-5ec9-4f8d-a8f2-10e0addd1b67\") " pod="openshift-ingress-canary/ingress-canary-9c99h" Nov 03 20:47:32.213817 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.213803 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7262fd-b29d-4c72-b717-ef9742bdd2db-config-volume\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:32.226314 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.226287 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvp8m\" (UniqueName: \"kubernetes.io/projected/542dd56a-cfab-40e2-a01b-a172651d46f8-kube-api-access-cvp8m\") pod \"iptables-alerter-wl57x\" (UID: \"542dd56a-cfab-40e2-a01b-a172651d46f8\") " pod="openshift-network-operator/iptables-alerter-wl57x" Nov 03 20:47:32.226439 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.226425 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkv6p\" (UniqueName: \"kubernetes.io/projected/5ea4f6f2-4699-46d7-91bc-8391dde191eb-kube-api-access-gkv6p\") pod \"kas-connection-checker-685895c7c9-kwzxh\" (UID: \"5ea4f6f2-4699-46d7-91bc-8391dde191eb\") " pod="kube-system/kas-connection-checker-685895c7c9-kwzxh" Nov 03 20:47:32.227086 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.227062 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mzl9\" (UniqueName: \"kubernetes.io/projected/e29718e2-f954-4ec8-9db2-d6c3638bc51c-kube-api-access-6mzl9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4\" (UID: \"e29718e2-f954-4ec8-9db2-d6c3638bc51c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" Nov 03 20:47:32.227169 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.227106 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhh48\" (UniqueName: \"kubernetes.io/projected/510fbc36-62de-4096-b8e8-d8606a4ad7d5-kube-api-access-fhh48\") pod \"service-ca-operator-544d749c8c-s82vc\" (UID: \"510fbc36-62de-4096-b8e8-d8606a4ad7d5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" Nov 03 20:47:32.231676 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.231645 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l7br\" (UniqueName: \"kubernetes.io/projected/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-api-access-8l7br\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:32.231769 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.231652 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zwts\" (UniqueName: \"kubernetes.io/projected/34ba944d-b672-489d-9acc-fde356202fbe-kube-api-access-6zwts\") pod \"kas-connection-checker-685895c7c9-ghxn6\" (UID: \"34ba944d-b672-489d-9acc-fde356202fbe\") " pod="kube-system/kas-connection-checker-685895c7c9-ghxn6" Nov 03 20:47:32.231769 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.231750 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hd42\" (UniqueName: \"kubernetes.io/projected/8d21a1ae-f712-4f89-b34f-8016fd918484-kube-api-access-2hd42\") pod \"insights-operator-7c4986d449-92hts\" (UID: \"8d21a1ae-f712-4f89-b34f-8016fd918484\") " pod="openshift-insights/insights-operator-7c4986d449-92hts" Nov 03 20:47:32.231952 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.231932 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbvs7\" (UniqueName: \"kubernetes.io/projected/ed5e1db5-564c-4abd-87c4-a2dda264a926-kube-api-access-fbvs7\") pod \"migrator-66df5d496b-qlrbs\" (UID: \"ed5e1db5-564c-4abd-87c4-a2dda264a926\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlrbs" Nov 03 20:47:32.231952 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.231945 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc6rc\" (UniqueName: \"kubernetes.io/projected/462e871a-1226-4f07-b34c-ef612fa6bf52-kube-api-access-pc6rc\") pod \"node-ca-r9lld\" (UID: \"462e871a-1226-4f07-b34c-ef612fa6bf52\") " pod="openshift-image-registry/node-ca-r9lld" Nov 03 20:47:32.232282 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.232264 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mzjv\" (UniqueName: \"kubernetes.io/projected/eafb0c22-9550-4b1c-8134-fc80562025eb-kube-api-access-2mzjv\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.232398 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.232379 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brl7b\" (UniqueName: \"kubernetes.io/projected/d0d6aa7f-3a2b-482f-8339-291a11a77c22-kube-api-access-brl7b\") pod \"network-check-source-6f78d59f99-t4tng\" (UID: \"d0d6aa7f-3a2b-482f-8339-291a11a77c22\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t4tng" Nov 03 20:47:32.232441 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.232393 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9dm2\" (UniqueName: \"kubernetes.io/projected/e863782c-c289-4fa5-93b1-b475358a5001-kube-api-access-l9dm2\") pod \"kas-connection-checker-685895c7c9-bd5l5\" (UID: \"e863782c-c289-4fa5-93b1-b475358a5001\") " pod="kube-system/kas-connection-checker-685895c7c9-bd5l5" Nov 03 20:47:32.232480 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.232458 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t42hf\" (UniqueName: \"kubernetes.io/projected/c7559e5a-a8fd-450c-8f40-f2072e93a9b5-kube-api-access-t42hf\") pod \"node-resolver-sm2mn\" (UID: \"c7559e5a-a8fd-450c-8f40-f2072e93a9b5\") " pod="openshift-dns/node-resolver-sm2mn" Nov 03 20:47:32.232599 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.232583 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m4g8\" (UniqueName: \"kubernetes.io/projected/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-kube-api-access-5m4g8\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:32.233351 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.233330 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nwhb\" (UniqueName: \"kubernetes.io/projected/ff7262fd-b29d-4c72-b717-ef9742bdd2db-kube-api-access-8nwhb\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:32.234344 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.234322 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxhlc\" (UniqueName: \"kubernetes.io/projected/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-kube-api-access-gxhlc\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.237231 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.236707 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbfzh\" (UniqueName: \"kubernetes.io/projected/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-kube-api-access-dbfzh\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:47:32.237231 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.236883 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-bound-sa-token\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:32.237231 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.237009 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltgln\" (UniqueName: \"kubernetes.io/projected/e3c555a1-e6bd-4042-be58-404cb6693922-kube-api-access-ltgln\") pod \"console-operator-7cd56b8dbc-ccxnp\" (UID: \"e3c555a1-e6bd-4042-be58-404cb6693922\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:32.237231 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.237032 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhvdt\" (UniqueName: \"kubernetes.io/projected/d2af25ca-9ad2-481c-addd-ebe3d999fc4e-kube-api-access-fhvdt\") pod \"migrator-66df5d496b-ndsnt\" (UID: \"d2af25ca-9ad2-481c-addd-ebe3d999fc4e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ndsnt" Nov 03 20:47:32.237231 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.237068 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vqts\" (UniqueName: \"kubernetes.io/projected/84556453-e5d9-4449-ba59-819c248e1e2a-kube-api-access-4vqts\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:32.237231 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.237196 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nd4m\" (UniqueName: \"kubernetes.io/projected/8f39698b-0204-4492-ac16-e1728e71b8d6-kube-api-access-8nd4m\") pod \"volume-data-source-validator-58c9f5f46d-nm757\" (UID: \"8f39698b-0204-4492-ac16-e1728e71b8d6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nm757" Nov 03 20:47:32.237515 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.237494 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cjnm\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-kube-api-access-5cjnm\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:32.237547 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.237512 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdvvx\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-kube-api-access-cdvvx\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:32.237547 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.237519 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bxrc\" (UniqueName: \"kubernetes.io/projected/12ce6198-3748-49b2-ad3d-f5c3d701fc96-kube-api-access-6bxrc\") pod \"service-ca-5fffc97bdf-vcjdr\" (UID: \"12ce6198-3748-49b2-ad3d-f5c3d701fc96\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" Nov 03 20:47:32.237603 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.237572 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7n4p\" (UniqueName: \"kubernetes.io/projected/8e456784-7491-426a-80e5-17a9dfa2f07f-kube-api-access-v7n4p\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.237633 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.237607 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76nsp\" (UniqueName: \"kubernetes.io/projected/245acdd3-cebd-49fe-bf1e-867df120ac8e-kube-api-access-76nsp\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.237681 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.237667 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd729\" (UniqueName: \"kubernetes.io/projected/af51f192-4170-4d3b-85ae-7d2ceb14de05-kube-api-access-rd729\") pod \"network-check-target-l7lxn\" (UID: \"af51f192-4170-4d3b-85ae-7d2ceb14de05\") " pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:32.238574 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.238554 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj7td\" (UniqueName: \"kubernetes.io/projected/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-kube-api-access-sj7td\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:32.238821 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.238802 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvs2\" (UniqueName: \"kubernetes.io/projected/24a97420-efcd-4a6f-b072-1e5792264188-kube-api-access-8zvs2\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:32.239552 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.239536 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq6p2\" (UniqueName: \"kubernetes.io/projected/1f33677f-ee87-4cd1-bdb0-b74000834db2-kube-api-access-wq6p2\") pod \"periodic-gathering-4dvbp-dpsms\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:47:32.241678 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.241662 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-bound-sa-token\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:32.244188 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.244172 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fflvq\" (UniqueName: \"kubernetes.io/projected/42d01b00-5ec9-4f8d-a8f2-10e0addd1b67-kube-api-access-fflvq\") pod \"ingress-canary-9c99h\" (UID: \"42d01b00-5ec9-4f8d-a8f2-10e0addd1b67\") " pod="openshift-ingress-canary/ingress-canary-9c99h" Nov 03 20:47:32.250157 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.250141 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sm2mn_c7559e5a-a8fd-450c-8f40-f2072e93a9b5/dns-node-resolver/0.log" Nov 03 20:47:32.273355 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273324 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-var-lib-cni-bin\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.273355 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273360 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/462e871a-1226-4f07-b34c-ef612fa6bf52-host\") pod \"node-ca-r9lld\" (UID: \"462e871a-1226-4f07-b34c-ef612fa6bf52\") " pod="openshift-image-registry/node-ca-r9lld" Nov 03 20:47:32.273577 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273378 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-run-openvswitch\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.273577 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273393 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-run-k8s-cni-cncf-io\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.273577 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273416 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-kubelet-config\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:32.273577 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273431 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.273577 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273445 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-var-lib-cni-bin\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.273577 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273462 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-run-openvswitch\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.273577 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273456 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/462e871a-1226-4f07-b34c-ef612fa6bf52-host\") pod \"node-ca-r9lld\" (UID: \"462e871a-1226-4f07-b34c-ef612fa6bf52\") " pod="openshift-image-registry/node-ca-r9lld" Nov 03 20:47:32.273577 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273489 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.273577 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273520 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-run-k8s-cni-cncf-io\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.273577 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273548 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-dbus\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:32.273577 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273571 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-system-cni-dir\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:47:32.273577 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273577 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-kubelet-config\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273588 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-socket-dir-parent\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273594 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f1eb3e05-8b22-4992-97d9-c71ecc6b99dc-dbus\") pod \"global-pull-secret-syncer-lvbhk\" (UID: \"f1eb3e05-8b22-4992-97d9-c71ecc6b99dc\") " pod="kube-system/global-pull-secret-syncer-lvbhk" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273603 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273635 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-system-cni-dir\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273644 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-socket-dir-parent\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273667 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.273687 4963 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273695 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-sysctl-d\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.273741 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls podName:ff7262fd-b29d-4c72-b717-ef9742bdd2db nodeName:}" failed. No retries permitted until 2026-11-03 20:47:32.773725871 +0000 UTC m=+1.933798156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls") pod "dns-default-njb7z" (UID: "ff7262fd-b29d-4c72-b717-ef9742bdd2db") : secret "dns-default-metrics-tls" not found Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.273765 4963 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.273779 4963 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687954fc7b-25pdn: secret "image-registry-tls" not found Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.273836 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls podName:15d37b2c-d051-419e-b0cb-d761cac6ff74 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:32.773823783 +0000 UTC m=+1.933896033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls") pod "image-registry-687954fc7b-25pdn" (UID: "15d37b2c-d051-419e-b0cb-d761cac6ff74") : secret "image-registry-tls" not found Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273832 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-sysctl-d\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273860 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-var-lib-kubelet\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273883 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/542dd56a-cfab-40e2-a01b-a172651d46f8-host-slash\") pod \"iptables-alerter-wl57x\" (UID: \"542dd56a-cfab-40e2-a01b-a172651d46f8\") " pod="openshift-network-operator/iptables-alerter-wl57x" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273927 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273951 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-var-lib-kubelet\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273952 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/542dd56a-cfab-40e2-a01b-a172651d46f8-host-slash\") pod \"iptables-alerter-wl57x\" (UID: \"542dd56a-cfab-40e2-a01b-a172651d46f8\") " pod="openshift-network-operator/iptables-alerter-wl57x" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273954 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7559e5a-a8fd-450c-8f40-f2072e93a9b5-hosts-file\") pod \"node-resolver-sm2mn\" (UID: \"c7559e5a-a8fd-450c-8f40-f2072e93a9b5\") " pod="openshift-dns/node-resolver-sm2mn" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273989 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7559e5a-a8fd-450c-8f40-f2072e93a9b5-hosts-file\") pod \"node-resolver-sm2mn\" (UID: \"c7559e5a-a8fd-450c-8f40-f2072e93a9b5\") " pod="openshift-dns/node-resolver-sm2mn" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.273988 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274003 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-systemd-units\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274030 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-run-netns\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274052 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-hostroot\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274081 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-sysctl-conf\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274100 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-run-netns\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274106 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-lib-modules\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274109 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-hostroot\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274080 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-systemd-units\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274131 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274160 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-cni-dir\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.274175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274185 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-var-lib-cni-multus\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274198 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-sysctl-conf\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274217 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-lib-modules\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274221 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274235 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-var-lib-cni-multus\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.274251 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:32.774237864 +0000 UTC m=+1.934310134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : configmap references non-existent config key: service-ca.crt Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274266 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-log-socket\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274252 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-cni-dir\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274285 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.274300 4963 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274314 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-log-socket\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274316 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274326 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274345 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-modprobe-d\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.274362 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:32.774346817 +0000 UTC m=+1.934419090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : secret "router-metrics-certs-default" not found Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.274375 4963 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.274387 4963 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75999699c7-pfc8g: secret "image-registry-tls" not found Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274392 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-kubelet\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.274421 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls podName:793c02dd-468b-4308-bac2-67d04e40579c nodeName:}" failed. No retries permitted until 2026-11-03 20:47:32.774408828 +0000 UTC m=+1.934481079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls") pod "image-registry-75999699c7-pfc8g" (UID: "793c02dd-468b-4308-bac2-67d04e40579c") : secret "image-registry-tls" not found Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274429 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-modprobe-d\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274436 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-run\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274439 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-kubelet\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274463 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274484 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-var-lib-kubelet\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274490 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-run\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274502 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274517 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-var-lib-kubelet\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.274523 4963 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274546 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-cnibin\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274524 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb02f36e-58c6-47fb-b6f7-0ba78d029ffb-cnibin\") pod \"multus-additional-cni-plugins-tgl9l\" (UID: \"fb02f36e-58c6-47fb-b6f7-0ba78d029ffb\") " pod="openshift-multus/multus-additional-cni-plugins-tgl9l" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274555 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.274564 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls podName:2955ecad-cad6-48d2-a2f0-1c1aae3d47f4 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:32.774551439 +0000 UTC m=+1.934623717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8xh9" (UID: "2955ecad-cad6-48d2-a2f0-1c1aae3d47f4") : secret "insights-runtime-extractor-tls" not found Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274598 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-slash\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274622 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-cni-netd\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274639 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-slash\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274648 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-host\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274663 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-cni-netd\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274672 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-var-lib-openvswitch\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274691 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-host\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274698 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-crio-socket\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274715 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-var-lib-openvswitch\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274722 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-run-ovn\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274745 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-crio-socket\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274748 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-conf-dir\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274772 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-etc-kubernetes\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274798 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274777 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-run-ovn\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274827 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274833 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-etc-kubernetes\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274858 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-multus-conf-dir\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274852 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-cnibin\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.274884 4963 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274895 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-run-multus-certs\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274928 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-cnibin\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274883 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274941 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-etc-openvswitch\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274946 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-host-run-multus-certs\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274962 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-etc-openvswitch\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.274968 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.274976 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs podName:84556453-e5d9-4449-ba59-819c248e1e2a nodeName:}" failed. No retries permitted until 2026-11-03 20:47:32.774967723 +0000 UTC m=+1.935039973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs") pod "network-metrics-daemon-mr55c" (UID: "84556453-e5d9-4449-ba59-819c248e1e2a") : secret "metrics-daemon-secret" not found Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275022 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-run-systemd\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275046 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275059 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-run-systemd\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275060 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-node-log\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275079 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-node-log\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275085 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-systemd\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275132 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-sysconfig\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275148 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-registration-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275163 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-systemd\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275172 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-run-netns\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275174 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-sysconfig\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275194 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-kubernetes\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275202 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-run-netns\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.275203 4963 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275217 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-registration-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275229 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-etc-kubernetes\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.275242 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls podName:4a6bf982-7a96-41ae-b24d-78cdabbb7c37 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:32.775234028 +0000 UTC m=+1.935306278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-72zbp" (UID: "4a6bf982-7a96-41ae-b24d-78cdabbb7c37") : secret "samples-operator-tls" not found Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275229 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-device-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275262 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-device-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275262 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-system-cni-dir\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275293 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ea105ed-0fa7-46a4-8d9f-94714d3a9b20-system-cni-dir\") pod \"multus-snszh\" (UID: \"3ea105ed-0fa7-46a4-8d9f-94714d3a9b20\") " pod="openshift-multus/multus-snszh" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275292 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xwt2z\" (UID: \"fb96b430-8876-41db-a35c-19db3cee7635\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275318 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-sys\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275332 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-socket-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.275340 4963 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275349 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-cni-bin\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275373 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-sys-fs\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.275385 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert podName:fb96b430-8876-41db-a35c-19db3cee7635 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:32.775372337 +0000 UTC m=+1.935444608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xwt2z" (UID: "fb96b430-8876-41db-a35c-19db3cee7635") : secret "networking-console-plugin-cert" not found Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275397 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eafb0c22-9550-4b1c-8134-fc80562025eb-sys\") pod \"tuned-7m7vq\" (UID: \"eafb0c22-9550-4b1c-8134-fc80562025eb\") " pod="openshift-cluster-node-tuning-operator/tuned-7m7vq" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275409 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275415 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-sys-fs\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275448 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/245acdd3-cebd-49fe-bf1e-867df120ac8e-socket-dir\") pod \"aws-ebs-csi-driver-node-pgxxn\" (UID: \"245acdd3-cebd-49fe-bf1e-867df120ac8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxxn" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275469 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e456784-7491-426a-80e5-17a9dfa2f07f-host-cni-bin\") pod \"ovnkube-node-vkh7b\" (UID: \"8e456784-7491-426a-80e5-17a9dfa2f07f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.275488 4963 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.275521 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls podName:24a97420-efcd-4a6f-b072-1e5792264188 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:32.775509755 +0000 UTC m=+1.935582010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gkklk" (UID: "24a97420-efcd-4a6f-b072-1e5792264188") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:47:32.275644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.275548 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:32.354259 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.354225 4963 scope.go:122] "RemoveContainer" containerID="29cf45ae230e23b9bed36cf8ecbd1b8c7222aa0b362a7fee1f51cdfc7b569516" Nov 03 20:47:32.370664 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.370571 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" Nov 03 20:47:32.370830 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.370811 4963 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:47:32.740272 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.740237 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-vcjdr"] Nov 03 20:47:32.747123 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:32.747064 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ce6198_3748_49b2_ad3d_f5c3d701fc96.slice/crio-b1b8318f3f683074ad8bbf106b3004498aad35336265cfe9adec7599ad3fb811 WatchSource:0}: Error finding container b1b8318f3f683074ad8bbf106b3004498aad35336265cfe9adec7599ad3fb811: Status 404 returned error can't find the container with id b1b8318f3f683074ad8bbf106b3004498aad35336265cfe9adec7599ad3fb811 Nov 03 20:47:32.780701 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.780672 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:32.780826 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.780735 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:32.780826 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.780779 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:32.780826 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.780806 4963 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:47:32.781015 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.780833 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xwt2z\" (UID: \"fb96b430-8876-41db-a35c-19db3cee7635\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:32.781015 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.780862 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls podName:2955ecad-cad6-48d2-a2f0-1c1aae3d47f4 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:33.780847937 +0000 UTC m=+2.940920212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8xh9" (UID: "2955ecad-cad6-48d2-a2f0-1c1aae3d47f4") : secret "insights-runtime-extractor-tls" not found Nov 03 20:47:32.781015 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.780880 4963 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:47:32.781015 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.780931 4963 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:47:32.781015 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.780952 4963 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:47:32.781015 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.780965 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls podName:4a6bf982-7a96-41ae-b24d-78cdabbb7c37 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:33.780948415 +0000 UTC m=+2.941020682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-72zbp" (UID: "4a6bf982-7a96-41ae-b24d-78cdabbb7c37") : secret "samples-operator-tls" not found Nov 03 20:47:32.781015 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.780993 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs podName:84556453-e5d9-4449-ba59-819c248e1e2a nodeName:}" failed. No retries permitted until 2026-11-03 20:47:33.780983433 +0000 UTC m=+2.941055887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs") pod "network-metrics-daemon-mr55c" (UID: "84556453-e5d9-4449-ba59-819c248e1e2a") : secret "metrics-daemon-secret" not found Nov 03 20:47:32.781015 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.780991 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:32.781015 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.781011 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert podName:fb96b430-8876-41db-a35c-19db3cee7635 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:33.781002387 +0000 UTC m=+2.941074650 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xwt2z" (UID: "fb96b430-8876-41db-a35c-19db3cee7635") : secret "networking-console-plugin-cert" not found Nov 03 20:47:32.781352 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.781054 4963 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:47:32.781352 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.781066 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:32.781352 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.781089 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls podName:24a97420-efcd-4a6f-b072-1e5792264188 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:33.78107891 +0000 UTC m=+2.941151178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gkklk" (UID: "24a97420-efcd-4a6f-b072-1e5792264188") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:47:32.781352 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.781108 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:32.781352 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.781120 4963 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:47:32.781352 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.781151 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls podName:ff7262fd-b29d-4c72-b717-ef9742bdd2db nodeName:}" failed. No retries permitted until 2026-11-03 20:47:33.7811427 +0000 UTC m=+2.941214951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls") pod "dns-default-njb7z" (UID: "ff7262fd-b29d-4c72-b717-ef9742bdd2db") : secret "dns-default-metrics-tls" not found Nov 03 20:47:32.781352 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.781197 4963 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:47:32.781352 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.781209 4963 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687954fc7b-25pdn: secret "image-registry-tls" not found Nov 03 20:47:32.781352 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.781241 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls podName:15d37b2c-d051-419e-b0cb-d761cac6ff74 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:33.781229578 +0000 UTC m=+2.941301830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls") pod "image-registry-687954fc7b-25pdn" (UID: "15d37b2c-d051-419e-b0cb-d761cac6ff74") : secret "image-registry-tls" not found Nov 03 20:47:32.781352 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.781269 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:32.781352 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.781302 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:32.781352 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:32.781337 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:32.781773 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.781364 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:33.781352842 +0000 UTC m=+2.941425092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : configmap references non-existent config key: service-ca.crt Nov 03 20:47:32.781773 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.781424 4963 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:47:32.781773 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.781435 4963 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75999699c7-pfc8g: secret "image-registry-tls" not found Nov 03 20:47:32.781773 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.781467 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls podName:793c02dd-468b-4308-bac2-67d04e40579c nodeName:}" failed. No retries permitted until 2026-11-03 20:47:33.781456929 +0000 UTC m=+2.941529194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls") pod "image-registry-75999699c7-pfc8g" (UID: "793c02dd-468b-4308-bac2-67d04e40579c") : secret "image-registry-tls" not found Nov 03 20:47:32.781773 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.781471 4963 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:47:32.781773 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:32.781501 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:33.78149215 +0000 UTC m=+2.941564404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : secret "router-metrics-certs-default" not found Nov 03 20:47:33.025345 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.025275 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 20:47:33.025749 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.025727 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/1.log" Nov 03 20:47:33.025794 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.025766 4963 generic.go:356] "Generic (PLEG): container finished" podID="e3c555a1-e6bd-4042-be58-404cb6693922" containerID="f92cbe5c0da1cef507ef817753f79983963992202df1a8acd3a08912c69c6f87" exitCode=255 Nov 03 20:47:33.025876 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.025853 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" event={"ID":"e3c555a1-e6bd-4042-be58-404cb6693922","Type":"ContainerDied","Data":"f92cbe5c0da1cef507ef817753f79983963992202df1a8acd3a08912c69c6f87"} Nov 03 20:47:33.025955 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.025942 4963 scope.go:122] "RemoveContainer" containerID="29cf45ae230e23b9bed36cf8ecbd1b8c7222aa0b362a7fee1f51cdfc7b569516" Nov 03 20:47:33.026322 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.026302 4963 scope.go:122] "RemoveContainer" containerID="f92cbe5c0da1cef507ef817753f79983963992202df1a8acd3a08912c69c6f87" Nov 03 20:47:33.026539 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.026518 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ccxnp_openshift-console-operator(e3c555a1-e6bd-4042-be58-404cb6693922)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" podUID="e3c555a1-e6bd-4042-be58-404cb6693922" Nov 03 20:47:33.027439 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.027412 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" event={"ID":"12ce6198-3748-49b2-ad3d-f5c3d701fc96","Type":"ContainerStarted","Data":"4828af75dcf22a76f273add643ebb4c58cd7aee7417c1ff85ef0f3afa8da84cc"} Nov 03 20:47:33.027561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.027454 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-vcjdr" event={"ID":"12ce6198-3748-49b2-ad3d-f5c3d701fc96","Type":"ContainerStarted","Data":"b1b8318f3f683074ad8bbf106b3004498aad35336265cfe9adec7599ad3fb811"} Nov 03 20:47:33.375237 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.375164 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r9lld_462e871a-1226-4f07-b34c-ef612fa6bf52/node-ca/0.log" Nov 03 20:47:33.775712 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.775672 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9c99h_42d01b00-5ec9-4f8d-a8f2-10e0addd1b67/serve-healthcheck-canary/0.log" Nov 03 20:47:33.791386 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.791358 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:33.791540 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.791407 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xwt2z\" (UID: \"fb96b430-8876-41db-a35c-19db3cee7635\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:33.791540 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.791442 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:33.791540 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.791486 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:33.791540 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.791509 4963 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:47:33.791769 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.791575 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls podName:4a6bf982-7a96-41ae-b24d-78cdabbb7c37 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:35.791560844 +0000 UTC m=+4.951633094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-72zbp" (UID: "4a6bf982-7a96-41ae-b24d-78cdabbb7c37") : secret "samples-operator-tls" not found Nov 03 20:47:33.791769 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.791511 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:33.791769 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.791586 4963 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:47:33.791769 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.791595 4963 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:47:33.791769 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.791610 4963 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687954fc7b-25pdn: secret "image-registry-tls" not found Nov 03 20:47:33.791769 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.791583 4963 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:47:33.791769 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.791659 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:33.791769 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.791509 4963 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:47:33.791769 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.791644 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls podName:ff7262fd-b29d-4c72-b717-ef9742bdd2db nodeName:}" failed. No retries permitted until 2026-11-03 20:47:35.791631858 +0000 UTC m=+4.951704108 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls") pod "dns-default-njb7z" (UID: "ff7262fd-b29d-4c72-b717-ef9742bdd2db") : secret "dns-default-metrics-tls" not found Nov 03 20:47:33.791769 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.791719 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls podName:15d37b2c-d051-419e-b0cb-d761cac6ff74 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:35.791706394 +0000 UTC m=+4.951778651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls") pod "image-registry-687954fc7b-25pdn" (UID: "15d37b2c-d051-419e-b0cb-d761cac6ff74") : secret "image-registry-tls" not found Nov 03 20:47:33.791769 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.791737 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls podName:24a97420-efcd-4a6f-b072-1e5792264188 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:35.791727486 +0000 UTC m=+4.951799736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gkklk" (UID: "24a97420-efcd-4a6f-b072-1e5792264188") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:47:33.791769 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.791757 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert podName:fb96b430-8876-41db-a35c-19db3cee7635 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:35.791748307 +0000 UTC m=+4.951820561 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xwt2z" (UID: "fb96b430-8876-41db-a35c-19db3cee7635") : secret "networking-console-plugin-cert" not found Nov 03 20:47:33.792307 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.791773 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:35.791765206 +0000 UTC m=+4.951837477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : configmap references non-existent config key: service-ca.crt Nov 03 20:47:33.792307 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.791818 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:33.792307 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.791873 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:33.792307 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.791938 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:33.792307 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.791944 4963 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:47:33.792307 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:33.791989 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:33.792307 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.792036 4963 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:47:33.792307 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.792045 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:35.792035762 +0000 UTC m=+4.952108013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : secret "router-metrics-certs-default" not found Nov 03 20:47:33.792307 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.791983 4963 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:47:33.792307 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.792071 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls podName:2955ecad-cad6-48d2-a2f0-1c1aae3d47f4 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:35.792059289 +0000 UTC m=+4.952131539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8xh9" (UID: "2955ecad-cad6-48d2-a2f0-1c1aae3d47f4") : secret "insights-runtime-extractor-tls" not found Nov 03 20:47:33.792307 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.792079 4963 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75999699c7-pfc8g: secret "image-registry-tls" not found Nov 03 20:47:33.792307 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.792098 4963 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:47:33.792307 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.792114 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls podName:793c02dd-468b-4308-bac2-67d04e40579c nodeName:}" failed. No retries permitted until 2026-11-03 20:47:35.792105073 +0000 UTC m=+4.952177337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls") pod "image-registry-75999699c7-pfc8g" (UID: "793c02dd-468b-4308-bac2-67d04e40579c") : secret "image-registry-tls" not found Nov 03 20:47:33.792307 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:33.792137 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs podName:84556453-e5d9-4449-ba59-819c248e1e2a nodeName:}" failed. No retries permitted until 2026-11-03 20:47:35.792125577 +0000 UTC m=+4.952197846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs") pod "network-metrics-daemon-mr55c" (UID: "84556453-e5d9-4449-ba59-819c248e1e2a") : secret "metrics-daemon-secret" not found Nov 03 20:47:34.041288 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:34.041226 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 20:47:34.091142 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:34.091114 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:34.276544 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:34.276510 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:34.375541 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:34.375447 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4_e29718e2-f954-4ec8-9db2-d6c3638bc51c/kube-storage-version-migrator-operator/0.log" Nov 03 20:47:34.618202 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:34.618168 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:34.810013 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:34.809977 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:35.046241 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.046205 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qlrbs" event={"ID":"ed5e1db5-564c-4abd-87c4-a2dda264a926","Type":"ContainerStarted","Data":"f760990b97c5eb23cabed5fd1b70d57cd3625cb2aab58c72cbc76b4967c70d49"} Nov 03 20:47:35.047707 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.047681 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ndsnt" event={"ID":"d2af25ca-9ad2-481c-addd-ebe3d999fc4e","Type":"ContainerStarted","Data":"666be30077bf54c497a974cb7f739090e9e935746ed59f2cb2dff7fe562bb27e"} Nov 03 20:47:35.302165 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.302126 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:47:35.490167 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.490134 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:35.813089 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.813051 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:35.813089 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.813087 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:35.813363 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.813121 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:35.813363 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.813141 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:35.813363 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.813168 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:35.813363 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.813192 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:35.813363 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:35.813220 4963 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:47:35.813363 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:35.813259 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:39.813238771 +0000 UTC m=+8.973311043 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : configmap references non-existent config key: service-ca.crt Nov 03 20:47:35.813363 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:35.813285 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls podName:ff7262fd-b29d-4c72-b717-ef9742bdd2db nodeName:}" failed. No retries permitted until 2026-11-03 20:47:39.813276015 +0000 UTC m=+8.973348265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls") pod "dns-default-njb7z" (UID: "ff7262fd-b29d-4c72-b717-ef9742bdd2db") : secret "dns-default-metrics-tls" not found Nov 03 20:47:35.813363 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.813225 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:35.813363 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:35.813307 4963 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:47:35.813363 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.813342 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:35.813363 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:35.813354 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:39.813339184 +0000 UTC m=+8.973411454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : secret "router-metrics-certs-default" not found Nov 03 20:47:35.813863 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.813405 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xwt2z\" (UID: \"fb96b430-8876-41db-a35c-19db3cee7635\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:35.813863 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:35.813416 4963 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:47:35.813863 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.813446 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:35.813863 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:35.813454 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls podName:4a6bf982-7a96-41ae-b24d-78cdabbb7c37 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:39.813443344 +0000 UTC m=+8.973515609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-72zbp" (UID: "4a6bf982-7a96-41ae-b24d-78cdabbb7c37") : secret "samples-operator-tls" not found Nov 03 20:47:35.813863 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.813470 4963 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 20:47:35.813863 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:35.813505 4963 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:47:35.813863 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:35.813544 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls podName:24a97420-efcd-4a6f-b072-1e5792264188 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:39.813531796 +0000 UTC m=+8.973604060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gkklk" (UID: "24a97420-efcd-4a6f-b072-1e5792264188") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:47:35.813863 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:35.813554 4963 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:47:35.813863 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:35.813582 4963 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:47:35.813863 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:35.813592 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls podName:2955ecad-cad6-48d2-a2f0-1c1aae3d47f4 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:39.813580009 +0000 UTC m=+8.973652299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8xh9" (UID: "2955ecad-cad6-48d2-a2f0-1c1aae3d47f4") : secret "insights-runtime-extractor-tls" not found Nov 03 20:47:35.813863 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:35.813683 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert podName:fb96b430-8876-41db-a35c-19db3cee7635 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:39.813661223 +0000 UTC m=+8.973733482 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-xwt2z" (UID: "fb96b430-8876-41db-a35c-19db3cee7635") : secret "networking-console-plugin-cert" not found Nov 03 20:47:35.815633 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.815614 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls\") pod \"image-registry-687954fc7b-25pdn\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:35.815725 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.815619 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84556453-e5d9-4449-ba59-819c248e1e2a-metrics-certs\") pod \"network-metrics-daemon-mr55c\" (UID: \"84556453-e5d9-4449-ba59-819c248e1e2a\") " pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:35.816049 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.816027 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls\") pod \"image-registry-75999699c7-pfc8g\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:35.865347 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.865312 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:35.955405 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.955371 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:35.965728 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.965699 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr55c" Nov 03 20:47:35.966851 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:35.966829 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:36.050595 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:36.050492 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:36.231528 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:36.231497 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:36.399554 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:36.399476 4963 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:36.399887 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:36.399873 4963 scope.go:122] "RemoveContainer" containerID="f92cbe5c0da1cef507ef817753f79983963992202df1a8acd3a08912c69c6f87" Nov 03 20:47:36.400106 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:36.400087 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ccxnp_openshift-console-operator(e3c555a1-e6bd-4042-be58-404cb6693922)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" podUID="e3c555a1-e6bd-4042-be58-404cb6693922" Nov 03 20:47:36.417441 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:36.417415 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:36.435967 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:36.435934 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mr55c"] Nov 03 20:47:36.441152 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:36.441130 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75999699c7-pfc8g"] Nov 03 20:47:36.442918 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:36.442882 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84556453_e5d9_4449_ba59_819c248e1e2a.slice/crio-906628399a581b6851b3566c8bddcd898ddb9c67d26051d579faec5cd2048e4f WatchSource:0}: Error finding container 906628399a581b6851b3566c8bddcd898ddb9c67d26051d579faec5cd2048e4f: Status 404 returned error can't find the container with id 906628399a581b6851b3566c8bddcd898ddb9c67d26051d579faec5cd2048e4f Nov 03 20:47:36.447941 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:36.447898 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793c02dd_468b_4308_bac2_67d04e40579c.slice/crio-7bd03ca88ca44079a345b0091fbee9fcfe884e6f39f4980f17d2b8df25e6e4da WatchSource:0}: Error finding container 7bd03ca88ca44079a345b0091fbee9fcfe884e6f39f4980f17d2b8df25e6e4da: Status 404 returned error can't find the container with id 7bd03ca88ca44079a345b0091fbee9fcfe884e6f39f4980f17d2b8df25e6e4da Nov 03 20:47:36.491554 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:36.491526 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-687954fc7b-25pdn"] Nov 03 20:47:36.499650 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:36.499622 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d37b2c_d051_419e_b0cb_d761cac6ff74.slice/crio-9617893a17047e6dfb9074088f3df4d922396ca26a9a4b5a69e058bbaaa1d366 WatchSource:0}: Error finding container 9617893a17047e6dfb9074088f3df4d922396ca26a9a4b5a69e058bbaaa1d366: Status 404 returned error can't find the container with id 9617893a17047e6dfb9074088f3df4d922396ca26a9a4b5a69e058bbaaa1d366 Nov 03 20:47:36.696006 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:36.695971 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:37.060509 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:37.060419 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75999699c7-pfc8g" event={"ID":"793c02dd-468b-4308-bac2-67d04e40579c","Type":"ContainerStarted","Data":"1d773514273d926767cc967482177472955d431d3ce61fac5ee41fdcb531c10d"} Nov 03 20:47:37.060509 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:37.060479 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:47:37.060509 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:37.060496 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75999699c7-pfc8g" event={"ID":"793c02dd-468b-4308-bac2-67d04e40579c","Type":"ContainerStarted","Data":"7bd03ca88ca44079a345b0091fbee9fcfe884e6f39f4980f17d2b8df25e6e4da"} Nov 03 20:47:37.062073 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:37.062046 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687954fc7b-25pdn" event={"ID":"15d37b2c-d051-419e-b0cb-d761cac6ff74","Type":"ContainerStarted","Data":"b1104829bd5d7d67b2cd71bc0b0788de9a9029df3a2ceb54f4d3b6b80f634f4e"} Nov 03 20:47:37.062195 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:37.062088 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687954fc7b-25pdn" event={"ID":"15d37b2c-d051-419e-b0cb-d761cac6ff74","Type":"ContainerStarted","Data":"9617893a17047e6dfb9074088f3df4d922396ca26a9a4b5a69e058bbaaa1d366"} Nov 03 20:47:37.063214 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:37.063189 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mr55c" event={"ID":"84556453-e5d9-4449-ba59-819c248e1e2a","Type":"ContainerStarted","Data":"906628399a581b6851b3566c8bddcd898ddb9c67d26051d579faec5cd2048e4f"} Nov 03 20:47:37.494522 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:37.494484 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:37.699580 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:37.699541 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:37.785926 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:37.785823 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j5mgz" Nov 03 20:47:38.067086 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:38.066998 4963 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:47:38.092634 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:38.092600 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:38.278361 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:38.278316 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:38.370862 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:38.370828 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:38.461008 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:38.460971 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:38.890789 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:38.890756 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:39.072007 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.071971 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:39.073147 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.073121 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mr55c" event={"ID":"84556453-e5d9-4449-ba59-819c248e1e2a","Type":"ContainerStarted","Data":"407ed450203b97ca0cc7cb7f6cda4106466558646e30f421c35a1c823f9e4e6d"} Nov 03 20:47:39.073263 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.073176 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mr55c" event={"ID":"84556453-e5d9-4449-ba59-819c248e1e2a","Type":"ContainerStarted","Data":"8bec774079e8dcfbaef74a8bff0515c258d72d721fbf786a615016550fa778bd"} Nov 03 20:47:39.444366 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.444317 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:47:39.490314 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.490274 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:39.696951 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.696860 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:39.857664 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.857619 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:39.857859 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.857674 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xwt2z\" (UID: \"fb96b430-8876-41db-a35c-19db3cee7635\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:39.857859 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.857709 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:39.857859 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.857748 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:39.857859 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.857829 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:39.858089 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.857879 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:39.858089 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.857946 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:39.858089 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:39.857968 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:47.857946579 +0000 UTC m=+17.018018846 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : configmap references non-existent config key: service-ca.crt Nov 03 20:47:39.858241 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:39.858108 4963 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:47:39.858241 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:39.858189 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs podName:eb7b7d50-f5bf-45b4-acaa-f1bc19831287 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:47.858169951 +0000 UTC m=+17.018242219 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs") pod "router-default-7fb544c95-8798s" (UID: "eb7b7d50-f5bf-45b4-acaa-f1bc19831287") : secret "router-metrics-certs-default" not found Nov 03 20:47:39.860315 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.860284 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2955ecad-cad6-48d2-a2f0-1c1aae3d47f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8xh9\" (UID: \"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4\") " pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:39.860421 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.860404 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff7262fd-b29d-4c72-b717-ef9742bdd2db-metrics-tls\") pod \"dns-default-njb7z\" (UID: \"ff7262fd-b29d-4c72-b717-ef9742bdd2db\") " pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:39.860421 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.860411 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24a97420-efcd-4a6f-b072-1e5792264188-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkklk\" (UID: \"24a97420-efcd-4a6f-b072-1e5792264188\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:39.860633 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.860615 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb96b430-8876-41db-a35c-19db3cee7635-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-xwt2z\" (UID: \"fb96b430-8876-41db-a35c-19db3cee7635\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:39.860743 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.860724 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a6bf982-7a96-41ae-b24d-78cdabbb7c37-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-72zbp\" (UID: \"4a6bf982-7a96-41ae-b24d-78cdabbb7c37\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:39.867543 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.867527 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:39.868619 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:39.868595 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d8xh9" Nov 03 20:47:40.093694 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:40.093605 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:40.154497 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:40.154462 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" Nov 03 20:47:40.156640 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:40.156616 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" Nov 03 20:47:40.161306 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:40.161281 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" Nov 03 20:47:40.240990 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:40.240884 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:47:40.241185 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:40.241093 4963 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:47:40.292119 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:40.292085 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:40.301806 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:40.301778 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7262fd_b29d_4c72_b717_ef9742bdd2db.slice/crio-d8a06800bd8a3e560a6208d131ad81cb17af55169e12d1ef98c8eec27b669284 WatchSource:0}: Error finding container d8a06800bd8a3e560a6208d131ad81cb17af55169e12d1ef98c8eec27b669284: Status 404 returned error can't find the container with id d8a06800bd8a3e560a6208d131ad81cb17af55169e12d1ef98c8eec27b669284 Nov 03 20:47:40.342280 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:40.342244 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-njb7z"] Nov 03 20:47:40.346380 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:40.346317 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d8xh9"] Nov 03 20:47:40.353877 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:40.353851 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2955ecad_cad6_48d2_a2f0_1c1aae3d47f4.slice/crio-4df65c24e3197864669655d8ce66eec6d0e3f445ea9352d0ca74cfc7f7e8f3cf WatchSource:0}: Error finding container 4df65c24e3197864669655d8ce66eec6d0e3f445ea9352d0ca74cfc7f7e8f3cf: Status 404 returned error can't find the container with id 4df65c24e3197864669655d8ce66eec6d0e3f445ea9352d0ca74cfc7f7e8f3cf Nov 03 20:47:40.530054 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:40.530019 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk"] Nov 03 20:47:40.534343 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:40.534318 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp"] Nov 03 20:47:40.537177 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:40.537145 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a97420_efcd_4a6f_b072_1e5792264188.slice/crio-e0bc42cac86f1cf9474c58ec4ef43172536596ce21a1bee537e06034d71e7a0d WatchSource:0}: Error finding container e0bc42cac86f1cf9474c58ec4ef43172536596ce21a1bee537e06034d71e7a0d: Status 404 returned error can't find the container with id e0bc42cac86f1cf9474c58ec4ef43172536596ce21a1bee537e06034d71e7a0d Nov 03 20:47:40.544753 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:40.544730 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-xwt2z"] Nov 03 20:47:40.552937 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:40.552893 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb96b430_8876_41db_a35c_19db3cee7635.slice/crio-5ef6dcb8c79a7fbf91abb67b4dfa8c6969f61038901118dfd4659fa7842b8af3 WatchSource:0}: Error finding container 5ef6dcb8c79a7fbf91abb67b4dfa8c6969f61038901118dfd4659fa7842b8af3: Status 404 returned error can't find the container with id 5ef6dcb8c79a7fbf91abb67b4dfa8c6969f61038901118dfd4659fa7842b8af3 Nov 03 20:47:40.594008 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:40.593979 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:40.594435 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:40.594419 4963 scope.go:122] "RemoveContainer" containerID="f92cbe5c0da1cef507ef817753f79983963992202df1a8acd3a08912c69c6f87" Nov 03 20:47:40.594635 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:47:40.594617 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ccxnp_openshift-console-operator(e3c555a1-e6bd-4042-be58-404cb6693922)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" podUID="e3c555a1-e6bd-4042-be58-404cb6693922" Nov 03 20:47:40.893596 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:40.893561 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:41.082520 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:41.082485 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-njb7z" event={"ID":"ff7262fd-b29d-4c72-b717-ef9742bdd2db","Type":"ContainerStarted","Data":"d8a06800bd8a3e560a6208d131ad81cb17af55169e12d1ef98c8eec27b669284"} Nov 03 20:47:41.083544 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:41.083521 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" event={"ID":"24a97420-efcd-4a6f-b072-1e5792264188","Type":"ContainerStarted","Data":"e0bc42cac86f1cf9474c58ec4ef43172536596ce21a1bee537e06034d71e7a0d"} Nov 03 20:47:41.084680 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:41.084659 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" event={"ID":"4a6bf982-7a96-41ae-b24d-78cdabbb7c37","Type":"ContainerStarted","Data":"bca3e7836daef3f7c2106c73db3d47984d8c95618ef3d627146fee8786972dd8"} Nov 03 20:47:41.086189 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:41.086169 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d8xh9" event={"ID":"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4","Type":"ContainerStarted","Data":"303cb711f8881de6a90ab61ac255905134b6ead074dc1df4fcbff398d3519cce"} Nov 03 20:47:41.086288 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:41.086215 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d8xh9" event={"ID":"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4","Type":"ContainerStarted","Data":"4df65c24e3197864669655d8ce66eec6d0e3f445ea9352d0ca74cfc7f7e8f3cf"} Nov 03 20:47:41.087174 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:41.087155 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" event={"ID":"fb96b430-8876-41db-a35c-19db3cee7635","Type":"ContainerStarted","Data":"5ef6dcb8c79a7fbf91abb67b4dfa8c6969f61038901118dfd4659fa7842b8af3"} Nov 03 20:47:41.491638 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:41.491582 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:41.689204 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:41.689170 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:42.094864 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:42.094824 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d8xh9" event={"ID":"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4","Type":"ContainerStarted","Data":"54b8660c078a1fd1def5530dfe53b2314551581af02dea42cfda2f03bf0c94b1"} Nov 03 20:47:42.099181 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:42.099151 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:42.299306 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:42.299124 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:42.697816 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:42.697713 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:42.962572 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:42.962490 4963 scope.go:122] "RemoveContainer" containerID="f92cbe5c0da1cef507ef817753f79983963992202df1a8acd3a08912c69c6f87" Nov 03 20:47:43.007781 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:43.007750 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:43.102126 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:43.102094 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 20:47:43.102307 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:43.102181 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" event={"ID":"e3c555a1-e6bd-4042-be58-404cb6693922","Type":"ContainerStarted","Data":"61b2cf9a709660d348e4e1d4e366ae6c62f6122a3675b0ab1dba6882c98bd58d"} Nov 03 20:47:43.103127 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:43.103102 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:43.105150 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:43.105037 4963 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-ccxnp container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body= Nov 03 20:47:43.105150 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:43.105089 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" podUID="e3c555a1-e6bd-4042-be58-404cb6693922" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" Nov 03 20:47:43.107035 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:43.107010 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-xwt2z" event={"ID":"fb96b430-8876-41db-a35c-19db3cee7635","Type":"ContainerStarted","Data":"9d86f77e775f5324d0dc91d8987d1312c476c006b5fd20111333b7edd4564e5a"} Nov 03 20:47:43.494092 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:43.494061 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:43.897196 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:43.897158 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:44.087133 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:44.087049 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:44.114433 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:44.114388 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkklk" event={"ID":"24a97420-efcd-4a6f-b072-1e5792264188","Type":"ContainerStarted","Data":"7bd62fc0a865d52bc989f65ef219d27053852e3e8a07d9be4a395b6f56adc93f"} Nov 03 20:47:44.116959 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:44.116663 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" event={"ID":"4a6bf982-7a96-41ae-b24d-78cdabbb7c37","Type":"ContainerStarted","Data":"73d2a34ec350ecc2d61db82c0d6df091e9d505af5c11bd260de22a2c80ee1bd3"} Nov 03 20:47:44.116959 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:44.116716 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-72zbp" event={"ID":"4a6bf982-7a96-41ae-b24d-78cdabbb7c37","Type":"ContainerStarted","Data":"40c0da52fb28fdc66c9aa28ad337183cdcbf85e78ac7d5ba2e40d5717bddc578"} Nov 03 20:47:44.119335 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:44.119306 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-njb7z" event={"ID":"ff7262fd-b29d-4c72-b717-ef9742bdd2db","Type":"ContainerStarted","Data":"b222d9db90ec9480f2a377f44a385c5d9844eca4a57daa1d15ea092db491b940"} Nov 03 20:47:44.119466 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:44.119356 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-njb7z" event={"ID":"ff7262fd-b29d-4c72-b717-ef9742bdd2db","Type":"ContainerStarted","Data":"ac8b908c0bb15ac133e2fae82018384bef327a78884387ff8f874b1986aec97c"} Nov 03 20:47:44.125550 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:44.125529 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ccxnp" Nov 03 20:47:44.289925 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:44.289867 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:44.702249 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:44.702216 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:44.888936 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:44.888870 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:45.078785 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:45.078668 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:45.125211 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:45.125175 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d8xh9" event={"ID":"2955ecad-cad6-48d2-a2f0-1c1aae3d47f4","Type":"ContainerStarted","Data":"32e2656f3dc199aaba06f18187834fb7985e48bb01691f81898298ee2190a934"} Nov 03 20:47:45.125518 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:45.125501 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:45.261340 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:45.261310 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:45.441476 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:45.441376 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:45.623410 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:45.623371 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:45.902787 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:45.902755 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:46.087071 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:46.087038 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:46.268993 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:46.268948 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:47.936558 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:47.936518 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:47.936558 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:47.936560 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:47.937197 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:47.937177 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-service-ca-bundle\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:47.939027 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:47.939008 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb7b7d50-f5bf-45b4-acaa-f1bc19831287-metrics-certs\") pod \"router-default-7fb544c95-8798s\" (UID: \"eb7b7d50-f5bf-45b4-acaa-f1bc19831287\") " pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:47.956570 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:47.956537 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:48.379648 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:48.379617 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7fb544c95-8798s"] Nov 03 20:47:48.386625 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:48.386597 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7b7d50_f5bf_45b4_acaa_f1bc19831287.slice/crio-54b28d3bf2f851aa78b4e6b6a43281ee7ffa350ee859b1e41fca7f552c430b5a WatchSource:0}: Error finding container 54b28d3bf2f851aa78b4e6b6a43281ee7ffa350ee859b1e41fca7f552c430b5a: Status 404 returned error can't find the container with id 54b28d3bf2f851aa78b4e6b6a43281ee7ffa350ee859b1e41fca7f552c430b5a Nov 03 20:47:49.146988 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:49.146952 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7fb544c95-8798s" event={"ID":"eb7b7d50-f5bf-45b4-acaa-f1bc19831287","Type":"ContainerStarted","Data":"7aff6ef63bdb843d05f0ae81ded1c5d3e13c684907eaab33cdd6b19200042c0b"} Nov 03 20:47:49.147399 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:49.147010 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7fb544c95-8798s" event={"ID":"eb7b7d50-f5bf-45b4-acaa-f1bc19831287","Type":"ContainerStarted","Data":"54b28d3bf2f851aa78b4e6b6a43281ee7ffa350ee859b1e41fca7f552c430b5a"} Nov 03 20:47:49.266197 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:49.266162 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:49.450507 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:49.450482 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:49.957306 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:49.957270 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:49.959899 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:49.959878 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:50.153151 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:50.153115 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:50.154452 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:50.154417 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7fb544c95-8798s" Nov 03 20:47:50.918264 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:50.918232 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:51.113305 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.113272 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:51.180598 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.180519 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rc2h2"] Nov 03 20:47:51.185465 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.185426 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rc2h2" Nov 03 20:47:51.228178 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.228144 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-kjnz2"] Nov 03 20:47:51.231826 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.231801 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75999699c7-pfc8g"] Nov 03 20:47:51.231981 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.231944 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-kjnz2" Nov 03 20:47:51.237193 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.237152 4963 patch_prober.go:29] interesting pod/image-registry-75999699c7-pfc8g container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 20:47:51.237325 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.237245 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-75999699c7-pfc8g" podUID="793c02dd-468b-4308-bac2-67d04e40579c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:47:51.261587 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.261554 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1c004a42-94de-4929-8617-d6a95d382361-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-rc2h2\" (UID: \"1c004a42-94de-4929-8617-d6a95d382361\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rc2h2" Nov 03 20:47:51.261767 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.261612 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gmhc\" (UniqueName: \"kubernetes.io/projected/a2f8097c-8639-4428-b9ed-a9287f4cef4d-kube-api-access-2gmhc\") pod \"downloads-6dd789c8b7-kjnz2\" (UID: \"a2f8097c-8639-4428-b9ed-a9287f4cef4d\") " pod="openshift-console/downloads-6dd789c8b7-kjnz2" Nov 03 20:47:51.261767 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.261667 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2f8097c-8639-4428-b9ed-a9287f4cef4d-tmp\") pod \"downloads-6dd789c8b7-kjnz2\" (UID: \"a2f8097c-8639-4428-b9ed-a9287f4cef4d\") " pod="openshift-console/downloads-6dd789c8b7-kjnz2" Nov 03 20:47:51.274577 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.274537 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 20:47:51.274775 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.274583 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bvmzg\"" Nov 03 20:47:51.275639 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.275616 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-578d79c767-lbsk9"] Nov 03 20:47:51.279290 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.279266 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.294245 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.294208 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 20:47:51.294245 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.294208 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 20:47:51.295826 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.295805 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rc2h2"] Nov 03 20:47:51.327461 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.327429 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:51.338943 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.338888 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-kjnz2"] Nov 03 20:47:51.346350 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.346317 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-578d79c767-lbsk9"] Nov 03 20:47:51.349324 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.349299 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-9lv8b\"" Nov 03 20:47:51.362847 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.362818 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvhll\" (UniqueName: \"kubernetes.io/projected/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-kube-api-access-rvhll\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.363024 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.362854 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-image-registry-private-configuration\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.363024 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.362950 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1c004a42-94de-4929-8617-d6a95d382361-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-rc2h2\" (UID: \"1c004a42-94de-4929-8617-d6a95d382361\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rc2h2" Nov 03 20:47:51.363024 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.362976 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-trusted-ca\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.363130 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.363076 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-installation-pull-secrets\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.363168 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.363135 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-ca-trust-extracted\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.363204 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.363165 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gmhc\" (UniqueName: \"kubernetes.io/projected/a2f8097c-8639-4428-b9ed-a9287f4cef4d-kube-api-access-2gmhc\") pod \"downloads-6dd789c8b7-kjnz2\" (UID: \"a2f8097c-8639-4428-b9ed-a9287f4cef4d\") " pod="openshift-console/downloads-6dd789c8b7-kjnz2" Nov 03 20:47:51.363204 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.363184 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-registry-certificates\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.363296 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.363277 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-registry-tls\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.363345 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.363327 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-bound-sa-token\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.363412 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.363396 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2f8097c-8639-4428-b9ed-a9287f4cef4d-tmp\") pod \"downloads-6dd789c8b7-kjnz2\" (UID: \"a2f8097c-8639-4428-b9ed-a9287f4cef4d\") " pod="openshift-console/downloads-6dd789c8b7-kjnz2" Nov 03 20:47:51.363733 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.363712 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2f8097c-8639-4428-b9ed-a9287f4cef4d-tmp\") pod \"downloads-6dd789c8b7-kjnz2\" (UID: \"a2f8097c-8639-4428-b9ed-a9287f4cef4d\") " pod="openshift-console/downloads-6dd789c8b7-kjnz2" Nov 03 20:47:51.365464 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.365430 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1c004a42-94de-4929-8617-d6a95d382361-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-rc2h2\" (UID: \"1c004a42-94de-4929-8617-d6a95d382361\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rc2h2" Nov 03 20:47:51.422432 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.422397 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gmhc\" (UniqueName: \"kubernetes.io/projected/a2f8097c-8639-4428-b9ed-a9287f4cef4d-kube-api-access-2gmhc\") pod \"downloads-6dd789c8b7-kjnz2\" (UID: \"a2f8097c-8639-4428-b9ed-a9287f4cef4d\") " pod="openshift-console/downloads-6dd789c8b7-kjnz2" Nov 03 20:47:51.464114 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.464084 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-installation-pull-secrets\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.464274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.464122 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-ca-trust-extracted\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.464274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.464139 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-registry-certificates\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.464274 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.464172 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-registry-tls\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.464434 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.464299 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-bound-sa-token\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.464434 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.464368 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvhll\" (UniqueName: \"kubernetes.io/projected/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-kube-api-access-rvhll\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.464434 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.464401 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-image-registry-private-configuration\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.464624 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.464438 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-trusted-ca\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.464624 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.464597 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-ca-trust-extracted\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.465254 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.465231 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-registry-certificates\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.465706 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.465688 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-trusted-ca\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.466941 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.466920 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-installation-pull-secrets\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.467197 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.467178 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-image-registry-private-configuration\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.467379 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.467358 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-registry-tls\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.500049 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.500015 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rc2h2" Nov 03 20:47:51.529645 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.529608 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-bound-sa-token\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.529791 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.529616 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvhll\" (UniqueName: \"kubernetes.io/projected/da3a5a52-8683-4bdb-b570-bd35a5bbf0af-kube-api-access-rvhll\") pod \"image-registry-578d79c767-lbsk9\" (UID: \"da3a5a52-8683-4bdb-b570-bd35a5bbf0af\") " pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.532321 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.532281 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:51.551010 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.550866 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-kjnz2" Nov 03 20:47:51.595074 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.595037 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:51.849992 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.849895 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:51.874085 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.874053 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rc2h2"] Nov 03 20:47:51.881614 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:51.881582 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c004a42_94de_4929_8617_d6a95d382361.slice/crio-8f3d00a070da007eeca6f80b797fd5de9275d28ca296635b8aea76fb37220470 WatchSource:0}: Error finding container 8f3d00a070da007eeca6f80b797fd5de9275d28ca296635b8aea76fb37220470: Status 404 returned error can't find the container with id 8f3d00a070da007eeca6f80b797fd5de9275d28ca296635b8aea76fb37220470 Nov 03 20:47:51.937482 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.937435 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-kjnz2"] Nov 03 20:47:51.945421 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:51.945389 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f8097c_8639_4428_b9ed_a9287f4cef4d.slice/crio-0519351bcf025fe293ceedbd643d7e8bd39988d6699c9b5ed15b284dc9adbebd WatchSource:0}: Error finding container 0519351bcf025fe293ceedbd643d7e8bd39988d6699c9b5ed15b284dc9adbebd: Status 404 returned error can't find the container with id 0519351bcf025fe293ceedbd643d7e8bd39988d6699c9b5ed15b284dc9adbebd Nov 03 20:47:51.990375 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:51.990339 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-578d79c767-lbsk9"] Nov 03 20:47:51.997271 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:47:51.997240 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3a5a52_8683_4bdb_b570_bd35a5bbf0af.slice/crio-98665f1edd985ed62259a633699160cb4951c46d3fb5938be6f59d33f6493385 WatchSource:0}: Error finding container 98665f1edd985ed62259a633699160cb4951c46d3fb5938be6f59d33f6493385: Status 404 returned error can't find the container with id 98665f1edd985ed62259a633699160cb4951c46d3fb5938be6f59d33f6493385 Nov 03 20:47:52.034962 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:52.034936 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:52.164064 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:52.163973 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-578d79c767-lbsk9" event={"ID":"da3a5a52-8683-4bdb-b570-bd35a5bbf0af","Type":"ContainerStarted","Data":"ed385979cb3c3778b00f323b085d16c3203b9bd2c86449e4769028c77b85e102"} Nov 03 20:47:52.164064 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:52.164042 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-578d79c767-lbsk9" event={"ID":"da3a5a52-8683-4bdb-b570-bd35a5bbf0af","Type":"ContainerStarted","Data":"98665f1edd985ed62259a633699160cb4951c46d3fb5938be6f59d33f6493385"} Nov 03 20:47:52.164300 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:52.164087 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:47:52.168477 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:52.168447 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-kjnz2" event={"ID":"a2f8097c-8639-4428-b9ed-a9287f4cef4d","Type":"ContainerStarted","Data":"0519351bcf025fe293ceedbd643d7e8bd39988d6699c9b5ed15b284dc9adbebd"} Nov 03 20:47:52.170156 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:52.170127 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rc2h2" event={"ID":"1c004a42-94de-4929-8617-d6a95d382361","Type":"ContainerStarted","Data":"8f3d00a070da007eeca6f80b797fd5de9275d28ca296635b8aea76fb37220470"} Nov 03 20:47:52.280576 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:52.280497 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-578d79c767-lbsk9" podStartSLOduration=1.280482296 podStartE2EDuration="1.280482296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:52.279138062 +0000 UTC m=+21.439210335" watchObservedRunningTime="2026-11-03 20:47:52.280482296 +0000 UTC m=+21.440554609" Nov 03 20:47:52.978844 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:52.978813 4963 patch_prober.go:29] interesting pod/image-registry-687954fc7b-25pdn container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 20:47:52.979032 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:52.978864 4963 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-687954fc7b-25pdn" podUID="15d37b2c-d051-419e-b0cb-d761cac6ff74" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:47:54.181567 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:54.181524 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rc2h2" event={"ID":"1c004a42-94de-4929-8617-d6a95d382361","Type":"ContainerStarted","Data":"f722fda4bfc1538ef3e3635ba89474b975832349abba95e6af35a638231232d1"} Nov 03 20:47:54.182090 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:54.181766 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rc2h2" Nov 03 20:47:54.187664 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:54.187636 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rc2h2" Nov 03 20:47:54.291920 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:54.291845 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rc2h2" podStartSLOduration=1.589192765 podStartE2EDuration="3.29182818s" totalImagesPullingTime="1.702635415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:54.291707266 +0000 UTC m=+23.451779539" watchObservedRunningTime="2026-11-03 20:47:54.29182818 +0000 UTC m=+23.451900453" Nov 03 20:47:54.867631 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:54.867599 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:55.049226 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:55.049187 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:55.131630 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:55.131541 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-njb7z" Nov 03 20:47:55.268602 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:55.268565 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:55.455739 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:55.455698 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:55.642248 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:55.642212 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:55.924142 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:55.924060 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:59.454225 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:47:59.449727 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:48:00.110438 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:00.110401 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:00.291260 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:00.291225 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:00.480977 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:00.480944 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:01.159885 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:01.159851 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:01.237509 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:01.237473 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:48:01.347292 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:01.347259 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:02.036783 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:02.036752 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkh7b" Nov 03 20:48:02.480088 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:02.480035 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:02.668498 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:02.668435 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:02.908314 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:02.908226 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:03.108030 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.107977 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:03.205201 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.205166 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wdvlq"] Nov 03 20:48:03.232361 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.232324 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.276706 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.276671 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/184db55a-c6cf-4e61-aff2-2231a47934f6-node-exporter-tls\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.276929 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.276734 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/184db55a-c6cf-4e61-aff2-2231a47934f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.276929 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.276840 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6z56\" (UniqueName: \"kubernetes.io/projected/184db55a-c6cf-4e61-aff2-2231a47934f6-kube-api-access-r6z56\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.276929 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.276886 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/184db55a-c6cf-4e61-aff2-2231a47934f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.277113 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.276942 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/184db55a-c6cf-4e61-aff2-2231a47934f6-node-exporter-textfile\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.277113 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.276974 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/184db55a-c6cf-4e61-aff2-2231a47934f6-node-exporter-wtmp\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.277113 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.277017 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/184db55a-c6cf-4e61-aff2-2231a47934f6-sys\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.277113 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.277051 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/184db55a-c6cf-4e61-aff2-2231a47934f6-root\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.277113 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.277101 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/184db55a-c6cf-4e61-aff2-2231a47934f6-metrics-client-ca\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.298399 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.298357 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:03.339808 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.339774 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 20:48:03.340107 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.340084 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-r6h6b\"" Nov 03 20:48:03.340184 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.340173 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 20:48:03.340251 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.340223 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 20:48:03.340311 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.340287 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 20:48:03.378374 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.378336 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/184db55a-c6cf-4e61-aff2-2231a47934f6-node-exporter-textfile\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.378374 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.378379 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/184db55a-c6cf-4e61-aff2-2231a47934f6-node-exporter-wtmp\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.378622 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.378459 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/184db55a-c6cf-4e61-aff2-2231a47934f6-sys\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.378622 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.378500 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/184db55a-c6cf-4e61-aff2-2231a47934f6-root\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.378622 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.378522 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/184db55a-c6cf-4e61-aff2-2231a47934f6-sys\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.378622 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.378534 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/184db55a-c6cf-4e61-aff2-2231a47934f6-node-exporter-wtmp\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.378622 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.378547 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/184db55a-c6cf-4e61-aff2-2231a47934f6-metrics-client-ca\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.378622 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.378566 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/184db55a-c6cf-4e61-aff2-2231a47934f6-root\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.378622 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.378593 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/184db55a-c6cf-4e61-aff2-2231a47934f6-node-exporter-tls\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.378856 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.378634 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/184db55a-c6cf-4e61-aff2-2231a47934f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.378856 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.378667 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6z56\" (UniqueName: \"kubernetes.io/projected/184db55a-c6cf-4e61-aff2-2231a47934f6-kube-api-access-r6z56\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.378856 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.378691 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/184db55a-c6cf-4e61-aff2-2231a47934f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.378856 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.378755 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/184db55a-c6cf-4e61-aff2-2231a47934f6-node-exporter-textfile\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.379147 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.379124 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/184db55a-c6cf-4e61-aff2-2231a47934f6-metrics-client-ca\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.379230 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.379211 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/184db55a-c6cf-4e61-aff2-2231a47934f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.381377 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.381316 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/184db55a-c6cf-4e61-aff2-2231a47934f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.381595 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.381400 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/184db55a-c6cf-4e61-aff2-2231a47934f6-node-exporter-tls\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.437666 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.437631 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6z56\" (UniqueName: \"kubernetes.io/projected/184db55a-c6cf-4e61-aff2-2231a47934f6-kube-api-access-r6z56\") pod \"node-exporter-wdvlq\" (UID: \"184db55a-c6cf-4e61-aff2-2231a47934f6\") " pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.482168 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.482078 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:03.612717 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.612678 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wdvlq" Nov 03 20:48:03.649305 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:48:03.649269 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184db55a_c6cf_4e61_aff2_2231a47934f6.slice/crio-466d8f1cf92617b2768e637cde7c68dcca0435caf220fdba38a41c59de407237 WatchSource:0}: Error finding container 466d8f1cf92617b2768e637cde7c68dcca0435caf220fdba38a41c59de407237: Status 404 returned error can't find the container with id 466d8f1cf92617b2768e637cde7c68dcca0435caf220fdba38a41c59de407237 Nov 03 20:48:03.672215 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.672170 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:03.855540 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:03.855463 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:04.231507 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:04.231467 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wdvlq" event={"ID":"184db55a-c6cf-4e61-aff2-2231a47934f6","Type":"ContainerStarted","Data":"466d8f1cf92617b2768e637cde7c68dcca0435caf220fdba38a41c59de407237"} Nov 03 20:48:04.631924 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:04.631811 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:04.839890 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:04.839849 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:05.305712 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:05.305674 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-l7lxn" Nov 03 20:48:05.481072 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:05.481036 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:05.870683 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:05.870644 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:06.054815 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:06.054782 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:06.243686 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:06.243650 4963 generic.go:356] "Generic (PLEG): container finished" podID="184db55a-c6cf-4e61-aff2-2231a47934f6" containerID="68f5339b5c06291aeac43c482928fe0a202078e505883ce6801010f305a0e6fd" exitCode=0 Nov 03 20:48:06.243871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:06.243748 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wdvlq" event={"ID":"184db55a-c6cf-4e61-aff2-2231a47934f6","Type":"ContainerDied","Data":"68f5339b5c06291aeac43c482928fe0a202078e505883ce6801010f305a0e6fd"} Nov 03 20:48:07.250827 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:07.250787 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wdvlq" event={"ID":"184db55a-c6cf-4e61-aff2-2231a47934f6","Type":"ContainerStarted","Data":"654a08d6f9706508b66838bd0c9db23b5ce8553b1be38da0777125d1752961e2"} Nov 03 20:48:07.251270 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:07.250851 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wdvlq" event={"ID":"184db55a-c6cf-4e61-aff2-2231a47934f6","Type":"ContainerStarted","Data":"558565cd428f9ee34b8141ce8a795e7e5ed5022a35af2d8250307c5e122d1cb1"} Nov 03 20:48:07.403448 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:07.403415 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:07.584096 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:07.584016 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:07.777394 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:07.777360 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:07.969007 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:07.968974 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:08.154420 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:08.154386 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:09.262821 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:09.262782 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-kjnz2" event={"ID":"a2f8097c-8639-4428-b9ed-a9287f4cef4d","Type":"ContainerStarted","Data":"bd5ce9752bae32cbeb22f2ad8a5ce9e04e74375029c2605dc1342ca15ed43630"} Nov 03 20:48:09.263359 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:09.263131 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-kjnz2" Nov 03 20:48:09.277449 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:09.277410 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-kjnz2" Nov 03 20:48:09.469686 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:09.469627 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wdvlq" podStartSLOduration=4.152597983 podStartE2EDuration="6.469605336s" totalImagesPullingTime="2.220750654s" totalInitContainerRuntime="96.256699ms" isStatefulPod=true podCreationTimestamp="2026-11-03 20:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:07.411676701 +0000 UTC m=+36.571748972" watchObservedRunningTime="2026-11-03 20:48:09.469605336 +0000 UTC m=+38.629677612" Nov 03 20:48:09.470356 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:09.470314 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-kjnz2" podStartSLOduration=2.193964988 podStartE2EDuration="18.470300687s" totalImagesPullingTime="16.276335699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:09.421695919 +0000 UTC m=+38.581768192" watchObservedRunningTime="2026-11-03 20:48:09.470300687 +0000 UTC m=+38.630372958" Nov 03 20:48:12.341625 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:12.341527 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:12.533121 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:48:12.533079 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:48:12.536370 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:12.536344 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:13.180475 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:13.180442 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-578d79c767-lbsk9" Nov 03 20:48:13.395160 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:13.395006 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-687954fc7b-25pdn"] Nov 03 20:48:13.520087 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:13.520050 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:13.707130 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:13.707056 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:13.893385 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:13.893302 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:15.809611 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:15.809570 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:16.005692 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.005654 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:16.254390 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.254343 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75999699c7-pfc8g" podUID="793c02dd-468b-4308-bac2-67d04e40579c" containerName="registry" containerID="cri-o://1d773514273d926767cc967482177472955d431d3ce61fac5ee41fdcb531c10d" gracePeriod=30 Nov 03 20:48:16.570374 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.570339 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:48:16.709776 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.709734 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls\") pod \"793c02dd-468b-4308-bac2-67d04e40579c\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " Nov 03 20:48:16.709988 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.709809 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-image-registry-private-configuration\") pod \"793c02dd-468b-4308-bac2-67d04e40579c\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " Nov 03 20:48:16.709988 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.709878 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793c02dd-468b-4308-bac2-67d04e40579c-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/793c02dd-468b-4308-bac2-67d04e40579c-ca-trust-extracted\") pod \"793c02dd-468b-4308-bac2-67d04e40579c\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " Nov 03 20:48:16.709988 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.709982 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-bound-sa-token\") pod \"793c02dd-468b-4308-bac2-67d04e40579c\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " Nov 03 20:48:16.710404 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.710376 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-kube-api-access-5cjnm\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-kube-api-access-5cjnm\") pod \"793c02dd-468b-4308-bac2-67d04e40579c\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " Nov 03 20:48:16.710811 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.710790 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-registry-certificates\") pod \"793c02dd-468b-4308-bac2-67d04e40579c\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " Nov 03 20:48:16.711044 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.711024 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-installation-pull-secrets\") pod \"793c02dd-468b-4308-bac2-67d04e40579c\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " Nov 03 20:48:16.711290 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.711273 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-trusted-ca\") pod \"793c02dd-468b-4308-bac2-67d04e40579c\" (UID: \"793c02dd-468b-4308-bac2-67d04e40579c\") " Nov 03 20:48:16.711498 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.711267 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-registry-certificates" pod "793c02dd-468b-4308-bac2-67d04e40579c" (UID: "793c02dd-468b-4308-bac2-67d04e40579c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:16.711717 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.711692 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-trusted-ca" pod "793c02dd-468b-4308-bac2-67d04e40579c" (UID: "793c02dd-468b-4308-bac2-67d04e40579c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:16.711958 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.711940 4963 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-registry-certificates\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:16.712069 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.712055 4963 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/793c02dd-468b-4308-bac2-67d04e40579c-trusted-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:16.713954 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.713867 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls" pod "793c02dd-468b-4308-bac2-67d04e40579c" (UID: "793c02dd-468b-4308-bac2-67d04e40579c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:16.713954 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.713927 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-bound-sa-token" pod "793c02dd-468b-4308-bac2-67d04e40579c" (UID: "793c02dd-468b-4308-bac2-67d04e40579c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:16.714177 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.714127 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-installation-pull-secrets" pod "793c02dd-468b-4308-bac2-67d04e40579c" (UID: "793c02dd-468b-4308-bac2-67d04e40579c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:48:16.714284 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.714253 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-image-registry-private-configuration" pod "793c02dd-468b-4308-bac2-67d04e40579c" (UID: "793c02dd-468b-4308-bac2-67d04e40579c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:48:16.714976 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.714944 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-kube-api-access-5cjnm" pod "793c02dd-468b-4308-bac2-67d04e40579c" (UID: "793c02dd-468b-4308-bac2-67d04e40579c"). InnerVolumeSpecName "kube-api-access-5cjnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:16.718979 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.718952 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793c02dd-468b-4308-bac2-67d04e40579c-ca-trust-extracted" pod "793c02dd-468b-4308-bac2-67d04e40579c" (UID: "793c02dd-468b-4308-bac2-67d04e40579c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:48:16.813054 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.812963 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cjnm\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-kube-api-access-5cjnm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:16.813054 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.813001 4963 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-installation-pull-secrets\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:16.813054 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.813014 4963 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-registry-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:16.813054 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.813027 4963 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/793c02dd-468b-4308-bac2-67d04e40579c-image-registry-private-configuration\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:16.813054 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.813041 4963 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/793c02dd-468b-4308-bac2-67d04e40579c-ca-trust-extracted\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:16.813054 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:16.813054 4963 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/793c02dd-468b-4308-bac2-67d04e40579c-bound-sa-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:17.301755 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:17.301722 4963 generic.go:356] "Generic (PLEG): container finished" podID="793c02dd-468b-4308-bac2-67d04e40579c" containerID="1d773514273d926767cc967482177472955d431d3ce61fac5ee41fdcb531c10d" exitCode=0 Nov 03 20:48:17.301956 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:17.301775 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75999699c7-pfc8g" Nov 03 20:48:17.301956 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:17.301800 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75999699c7-pfc8g" event={"ID":"793c02dd-468b-4308-bac2-67d04e40579c","Type":"ContainerDied","Data":"1d773514273d926767cc967482177472955d431d3ce61fac5ee41fdcb531c10d"} Nov 03 20:48:17.301956 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:17.301853 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75999699c7-pfc8g" event={"ID":"793c02dd-468b-4308-bac2-67d04e40579c","Type":"ContainerDied","Data":"7bd03ca88ca44079a345b0091fbee9fcfe884e6f39f4980f17d2b8df25e6e4da"} Nov 03 20:48:17.301956 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:17.301876 4963 scope.go:122] "RemoveContainer" containerID="1d773514273d926767cc967482177472955d431d3ce61fac5ee41fdcb531c10d" Nov 03 20:48:17.310616 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:17.310590 4963 scope.go:122] "RemoveContainer" containerID="1d773514273d926767cc967482177472955d431d3ce61fac5ee41fdcb531c10d" Nov 03 20:48:17.310983 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:48:17.310952 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d773514273d926767cc967482177472955d431d3ce61fac5ee41fdcb531c10d\": container with ID starting with 1d773514273d926767cc967482177472955d431d3ce61fac5ee41fdcb531c10d not found: ID does not exist" containerID="1d773514273d926767cc967482177472955d431d3ce61fac5ee41fdcb531c10d" Nov 03 20:48:17.311047 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:17.310995 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d773514273d926767cc967482177472955d431d3ce61fac5ee41fdcb531c10d"} err="failed to get container status \"1d773514273d926767cc967482177472955d431d3ce61fac5ee41fdcb531c10d\": rpc error: code = NotFound desc = could not find container \"1d773514273d926767cc967482177472955d431d3ce61fac5ee41fdcb531c10d\": container with ID starting with 1d773514273d926767cc967482177472955d431d3ce61fac5ee41fdcb531c10d not found: ID does not exist" Nov 03 20:48:17.507389 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:17.507343 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75999699c7-pfc8g"] Nov 03 20:48:17.554609 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:17.554527 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75999699c7-pfc8g"] Nov 03 20:48:18.965265 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:18.965231 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="793c02dd-468b-4308-bac2-67d04e40579c" path="/var/lib/kubelet/pods/793c02dd-468b-4308-bac2-67d04e40579c/volumes" Nov 03 20:48:20.251352 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:20.251313 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:20.442423 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:20.442392 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:20.687885 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:20.687804 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:20.877757 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:20.877726 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:21.294280 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:21.294232 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:21.507368 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:21.507308 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:22.372214 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:22.372172 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:22.564516 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:22.564484 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:23.478466 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:23.478434 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:23.662888 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:23.662852 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:25.603655 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:25.603619 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:25.791628 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:25.791586 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:26.262082 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:26.262052 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:26.448595 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:26.448560 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:30.778841 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:30.778799 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:30.964695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:30.964664 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:38.394750 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.394717 4963 generic.go:356] "Generic (PLEG): container finished" podID="e29718e2-f954-4ec8-9db2-d6c3638bc51c" containerID="473e049556b75a1207b4bf4b4307a644de5e9a451792c92b16e2df7426e2e82c" exitCode=0 Nov 03 20:48:38.395258 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.394789 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" event={"ID":"e29718e2-f954-4ec8-9db2-d6c3638bc51c","Type":"ContainerDied","Data":"473e049556b75a1207b4bf4b4307a644de5e9a451792c92b16e2df7426e2e82c"} Nov 03 20:48:38.395258 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.395239 4963 scope.go:122] "RemoveContainer" containerID="473e049556b75a1207b4bf4b4307a644de5e9a451792c92b16e2df7426e2e82c" Nov 03 20:48:38.419230 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.419193 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-687954fc7b-25pdn" podUID="15d37b2c-d051-419e-b0cb-d761cac6ff74" containerName="registry" containerID="cri-o://b1104829bd5d7d67b2cd71bc0b0788de9a9029df3a2ceb54f4d3b6b80f634f4e" gracePeriod=30 Nov 03 20:48:38.702781 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.702751 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:48:38.791306 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.791271 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-bound-sa-token\") pod \"15d37b2c-d051-419e-b0cb-d761cac6ff74\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " Nov 03 20:48:38.791306 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.791311 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-kube-api-access-cdvvx\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-kube-api-access-cdvvx\") pod \"15d37b2c-d051-419e-b0cb-d761cac6ff74\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " Nov 03 20:48:38.791536 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.791331 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-trusted-ca\") pod \"15d37b2c-d051-419e-b0cb-d761cac6ff74\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " Nov 03 20:48:38.791536 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.791358 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls\") pod \"15d37b2c-d051-419e-b0cb-d761cac6ff74\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " Nov 03 20:48:38.791536 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.791389 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-installation-pull-secrets\") pod \"15d37b2c-d051-419e-b0cb-d761cac6ff74\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " Nov 03 20:48:38.791536 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.791418 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-certificates\") pod \"15d37b2c-d051-419e-b0cb-d761cac6ff74\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " Nov 03 20:48:38.791536 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.791472 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-image-registry-private-configuration\") pod \"15d37b2c-d051-419e-b0cb-d761cac6ff74\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " Nov 03 20:48:38.791796 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.791542 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d37b2c-d051-419e-b0cb-d761cac6ff74-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/15d37b2c-d051-419e-b0cb-d761cac6ff74-ca-trust-extracted\") pod \"15d37b2c-d051-419e-b0cb-d761cac6ff74\" (UID: \"15d37b2c-d051-419e-b0cb-d761cac6ff74\") " Nov 03 20:48:38.791979 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.791887 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-trusted-ca" pod "15d37b2c-d051-419e-b0cb-d761cac6ff74" (UID: "15d37b2c-d051-419e-b0cb-d761cac6ff74"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:38.792111 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.792014 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-certificates" pod "15d37b2c-d051-419e-b0cb-d761cac6ff74" (UID: "15d37b2c-d051-419e-b0cb-d761cac6ff74"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:38.793983 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.793950 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-image-registry-private-configuration" pod "15d37b2c-d051-419e-b0cb-d761cac6ff74" (UID: "15d37b2c-d051-419e-b0cb-d761cac6ff74"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:48:38.794429 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.794115 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls" pod "15d37b2c-d051-419e-b0cb-d761cac6ff74" (UID: "15d37b2c-d051-419e-b0cb-d761cac6ff74"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:38.794429 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.794257 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-bound-sa-token" pod "15d37b2c-d051-419e-b0cb-d761cac6ff74" (UID: "15d37b2c-d051-419e-b0cb-d761cac6ff74"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:38.794429 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.794271 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-installation-pull-secrets" pod "15d37b2c-d051-419e-b0cb-d761cac6ff74" (UID: "15d37b2c-d051-419e-b0cb-d761cac6ff74"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:48:38.794429 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.794308 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-kube-api-access-cdvvx" pod "15d37b2c-d051-419e-b0cb-d761cac6ff74" (UID: "15d37b2c-d051-419e-b0cb-d761cac6ff74"). InnerVolumeSpecName "kube-api-access-cdvvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:38.798426 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.798395 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d37b2c-d051-419e-b0cb-d761cac6ff74-ca-trust-extracted" pod "15d37b2c-d051-419e-b0cb-d761cac6ff74" (UID: "15d37b2c-d051-419e-b0cb-d761cac6ff74"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:48:38.892790 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.892745 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdvvx\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-kube-api-access-cdvvx\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:38.892790 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.892782 4963 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-trusted-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:38.892790 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.892793 4963 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:38.892790 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.892803 4963 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-installation-pull-secrets\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:38.893170 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.892812 4963 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/15d37b2c-d051-419e-b0cb-d761cac6ff74-registry-certificates\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:38.893170 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.892822 4963 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/15d37b2c-d051-419e-b0cb-d761cac6ff74-image-registry-private-configuration\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:38.893170 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.892833 4963 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/15d37b2c-d051-419e-b0cb-d761cac6ff74-ca-trust-extracted\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:38.893170 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:38.892846 4963 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15d37b2c-d051-419e-b0cb-d761cac6ff74-bound-sa-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:39.401632 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:39.401595 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xlqb4" event={"ID":"e29718e2-f954-4ec8-9db2-d6c3638bc51c","Type":"ContainerStarted","Data":"0770660beadc2d7f27ba1d385f1034723344ff247ac4850dba896b243ae22360"} Nov 03 20:48:39.402863 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:39.402837 4963 generic.go:356] "Generic (PLEG): container finished" podID="15d37b2c-d051-419e-b0cb-d761cac6ff74" containerID="b1104829bd5d7d67b2cd71bc0b0788de9a9029df3a2ceb54f4d3b6b80f634f4e" exitCode=0 Nov 03 20:48:39.402979 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:39.402882 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687954fc7b-25pdn" Nov 03 20:48:39.402979 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:39.402899 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687954fc7b-25pdn" event={"ID":"15d37b2c-d051-419e-b0cb-d761cac6ff74","Type":"ContainerDied","Data":"b1104829bd5d7d67b2cd71bc0b0788de9a9029df3a2ceb54f4d3b6b80f634f4e"} Nov 03 20:48:39.402979 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:39.402954 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687954fc7b-25pdn" event={"ID":"15d37b2c-d051-419e-b0cb-d761cac6ff74","Type":"ContainerDied","Data":"9617893a17047e6dfb9074088f3df4d922396ca26a9a4b5a69e058bbaaa1d366"} Nov 03 20:48:39.402979 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:39.402973 4963 scope.go:122] "RemoveContainer" containerID="b1104829bd5d7d67b2cd71bc0b0788de9a9029df3a2ceb54f4d3b6b80f634f4e" Nov 03 20:48:39.433956 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:39.433935 4963 scope.go:122] "RemoveContainer" containerID="b1104829bd5d7d67b2cd71bc0b0788de9a9029df3a2ceb54f4d3b6b80f634f4e" Nov 03 20:48:39.434294 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:48:39.434271 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1104829bd5d7d67b2cd71bc0b0788de9a9029df3a2ceb54f4d3b6b80f634f4e\": container with ID starting with b1104829bd5d7d67b2cd71bc0b0788de9a9029df3a2ceb54f4d3b6b80f634f4e not found: ID does not exist" containerID="b1104829bd5d7d67b2cd71bc0b0788de9a9029df3a2ceb54f4d3b6b80f634f4e" Nov 03 20:48:39.434349 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:39.434304 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1104829bd5d7d67b2cd71bc0b0788de9a9029df3a2ceb54f4d3b6b80f634f4e"} err="failed to get container status \"b1104829bd5d7d67b2cd71bc0b0788de9a9029df3a2ceb54f4d3b6b80f634f4e\": rpc error: code = NotFound desc = could not find container \"b1104829bd5d7d67b2cd71bc0b0788de9a9029df3a2ceb54f4d3b6b80f634f4e\": container with ID starting with b1104829bd5d7d67b2cd71bc0b0788de9a9029df3a2ceb54f4d3b6b80f634f4e not found: ID does not exist" Nov 03 20:48:39.619714 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:39.619677 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-687954fc7b-25pdn"] Nov 03 20:48:39.667141 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:39.667058 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-687954fc7b-25pdn"] Nov 03 20:48:40.965603 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:40.965563 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15d37b2c-d051-419e-b0cb-d761cac6ff74" path="/var/lib/kubelet/pods/15d37b2c-d051-419e-b0cb-d761cac6ff74/volumes" Nov 03 20:48:42.417823 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:42.417731 4963 generic.go:356] "Generic (PLEG): container finished" podID="8d21a1ae-f712-4f89-b34f-8016fd918484" containerID="0626089d7672d3fc680a9f4a5533a7d2eeb10627594dde65ce8a9e9e0458e1fd" exitCode=0 Nov 03 20:48:42.417823 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:42.417803 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-92hts" event={"ID":"8d21a1ae-f712-4f89-b34f-8016fd918484","Type":"ContainerDied","Data":"0626089d7672d3fc680a9f4a5533a7d2eeb10627594dde65ce8a9e9e0458e1fd"} Nov 03 20:48:42.418305 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:42.418207 4963 scope.go:122] "RemoveContainer" containerID="0626089d7672d3fc680a9f4a5533a7d2eeb10627594dde65ce8a9e9e0458e1fd" Nov 03 20:48:42.529598 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:48:42.529544 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:48:43.423788 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:43.423756 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-92hts" event={"ID":"8d21a1ae-f712-4f89-b34f-8016fd918484","Type":"ContainerStarted","Data":"e565bd5da3079dd0a17cb5360fe52afe15fdfdd96f8b2c4d334160a733412c77"} Nov 03 20:48:43.441093 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:43.441059 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4dvbp-dpsms"] Nov 03 20:48:43.441288 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:43.441266 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-4dvbp-dpsms" podUID="1f33677f-ee87-4cd1-bdb0-b74000834db2" containerName="insights-gathering" containerID="cri-o://82867be803ea01339a9743835e27bce66d65f1aa03767dbabf0164a9be40d687" gracePeriod=30 Nov 03 20:48:43.736275 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:43.736241 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:48:43.831993 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:43.831942 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-trusted-ca-bundle\") pod \"1f33677f-ee87-4cd1-bdb0-b74000834db2\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " Nov 03 20:48:43.831993 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:43.831992 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f33677f-ee87-4cd1-bdb0-b74000834db2-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1f33677f-ee87-4cd1-bdb0-b74000834db2-archives-path\") pod \"1f33677f-ee87-4cd1-bdb0-b74000834db2\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " Nov 03 20:48:43.832199 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:43.832027 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-service-ca-bundle\") pod \"1f33677f-ee87-4cd1-bdb0-b74000834db2\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " Nov 03 20:48:43.832199 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:43.832072 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f33677f-ee87-4cd1-bdb0-b74000834db2-kube-api-access-wq6p2\" (UniqueName: \"kubernetes.io/projected/1f33677f-ee87-4cd1-bdb0-b74000834db2-kube-api-access-wq6p2\") pod \"1f33677f-ee87-4cd1-bdb0-b74000834db2\" (UID: \"1f33677f-ee87-4cd1-bdb0-b74000834db2\") " Nov 03 20:48:43.832366 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:43.832328 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-trusted-ca-bundle" pod "1f33677f-ee87-4cd1-bdb0-b74000834db2" (UID: "1f33677f-ee87-4cd1-bdb0-b74000834db2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:43.832542 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:43.832521 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f33677f-ee87-4cd1-bdb0-b74000834db2-archives-path" pod "1f33677f-ee87-4cd1-bdb0-b74000834db2" (UID: "1f33677f-ee87-4cd1-bdb0-b74000834db2"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:48:43.832615 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:43.832595 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-service-ca-bundle" pod "1f33677f-ee87-4cd1-bdb0-b74000834db2" (UID: "1f33677f-ee87-4cd1-bdb0-b74000834db2"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:43.834498 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:43.834470 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f33677f-ee87-4cd1-bdb0-b74000834db2-kube-api-access-wq6p2" pod "1f33677f-ee87-4cd1-bdb0-b74000834db2" (UID: "1f33677f-ee87-4cd1-bdb0-b74000834db2"). InnerVolumeSpecName "kube-api-access-wq6p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:43.932857 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:43.932816 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq6p2\" (UniqueName: \"kubernetes.io/projected/1f33677f-ee87-4cd1-bdb0-b74000834db2-kube-api-access-wq6p2\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:43.932857 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:43.932849 4963 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-trusted-ca-bundle\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:43.932857 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:43.932860 4963 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1f33677f-ee87-4cd1-bdb0-b74000834db2-archives-path\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:43.933115 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:43.932870 4963 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f33677f-ee87-4cd1-bdb0-b74000834db2-service-ca-bundle\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:44.429132 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:44.429097 4963 generic.go:356] "Generic (PLEG): container finished" podID="1f33677f-ee87-4cd1-bdb0-b74000834db2" containerID="82867be803ea01339a9743835e27bce66d65f1aa03767dbabf0164a9be40d687" exitCode=2 Nov 03 20:48:44.429538 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:44.429137 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4dvbp-dpsms" Nov 03 20:48:44.429538 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:44.429173 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4dvbp-dpsms" event={"ID":"1f33677f-ee87-4cd1-bdb0-b74000834db2","Type":"ContainerDied","Data":"82867be803ea01339a9743835e27bce66d65f1aa03767dbabf0164a9be40d687"} Nov 03 20:48:44.429538 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:44.429222 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4dvbp-dpsms" event={"ID":"1f33677f-ee87-4cd1-bdb0-b74000834db2","Type":"ContainerDied","Data":"c36c6a76d6d9bf65bd809f109b63aed4184fc1227bede75fa9d2ba65a9c1b170"} Nov 03 20:48:44.429538 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:44.429241 4963 scope.go:122] "RemoveContainer" containerID="82867be803ea01339a9743835e27bce66d65f1aa03767dbabf0164a9be40d687" Nov 03 20:48:44.438529 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:44.438491 4963 scope.go:122] "RemoveContainer" containerID="82867be803ea01339a9743835e27bce66d65f1aa03767dbabf0164a9be40d687" Nov 03 20:48:44.443214 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:48:44.443178 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82867be803ea01339a9743835e27bce66d65f1aa03767dbabf0164a9be40d687\": container with ID starting with 82867be803ea01339a9743835e27bce66d65f1aa03767dbabf0164a9be40d687 not found: ID does not exist" containerID="82867be803ea01339a9743835e27bce66d65f1aa03767dbabf0164a9be40d687" Nov 03 20:48:44.443333 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:44.443223 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82867be803ea01339a9743835e27bce66d65f1aa03767dbabf0164a9be40d687"} err="failed to get container status \"82867be803ea01339a9743835e27bce66d65f1aa03767dbabf0164a9be40d687\": rpc error: code = NotFound desc = could not find container \"82867be803ea01339a9743835e27bce66d65f1aa03767dbabf0164a9be40d687\": container with ID starting with 82867be803ea01339a9743835e27bce66d65f1aa03767dbabf0164a9be40d687 not found: ID does not exist" Nov 03 20:48:44.584166 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:44.584127 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4dvbp-dpsms"] Nov 03 20:48:44.631449 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:44.631411 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-4dvbp-dpsms"] Nov 03 20:48:44.965314 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:44.965274 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f33677f-ee87-4cd1-bdb0-b74000834db2" path="/var/lib/kubelet/pods/1f33677f-ee87-4cd1-bdb0-b74000834db2/volumes" Nov 03 20:48:46.870668 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:46.870615 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:47.055136 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:47.055095 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:50.865068 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:50.865018 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:51.049054 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:51.049020 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:52.886315 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:52.886284 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:53.089574 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:53.089540 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:53.273009 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:53.272972 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:53.462112 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:48:53.462077 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:03.508014 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:49:03.507975 4963 generic.go:356] "Generic (PLEG): container finished" podID="510fbc36-62de-4096-b8e8-d8606a4ad7d5" containerID="a562f4272a7eea23c0e885af23eb5031cb064fa5ec92af43e2d86743c1020203" exitCode=0 Nov 03 20:49:03.508422 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:49:03.508049 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" event={"ID":"510fbc36-62de-4096-b8e8-d8606a4ad7d5","Type":"ContainerDied","Data":"a562f4272a7eea23c0e885af23eb5031cb064fa5ec92af43e2d86743c1020203"} Nov 03 20:49:03.508422 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:49:03.508380 4963 scope.go:122] "RemoveContainer" containerID="a562f4272a7eea23c0e885af23eb5031cb064fa5ec92af43e2d86743c1020203" Nov 03 20:49:04.514482 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:49:04.514441 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s82vc" event={"ID":"510fbc36-62de-4096-b8e8-d8606a4ad7d5","Type":"ContainerStarted","Data":"aeae58b851de2d6cf0ca25f7a2eeff1cb4c22a1c255e4c51bb81edf29c68724b"} Nov 03 20:49:05.866295 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:49:05.866236 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:06.055285 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:49:06.055226 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:12.527953 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:49:12.527893 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:49:21.620863 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:49:21.620822 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:24.412224 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:49:24.412184 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:42.525514 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:49:42.525432 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:50:12.526879 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:50:12.526843 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:50:22.037879 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:50:22.037840 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:31.211613 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:50:31.211546 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:31.399031 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:50:31.398995 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:42.526047 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:50:42.526007 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:51:12.526723 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:51:12.526638 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:51:21.771320 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:51:21.771283 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:23.345240 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:51:23.345195 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:40.715230 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:51:40.715185 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:42.526326 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:51:42.526292 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:52:12.526418 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:52:12.526378 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:52:15.475283 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:15.475247 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:15.709542 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:15.709507 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:16.948871 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:16.948836 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:17.148852 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:17.148813 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:18.259004 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:18.258963 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:19.138636 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:19.138594 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:19.322333 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:19.322289 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:22.038542 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:22.038504 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:22.225068 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:22.225036 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:22.629815 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:22.629781 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:22.878581 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:22.878544 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:23.406885 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.406854 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:23.620074 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.620038 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v"] Nov 03 20:52:23.620361 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.620347 4963 cpu_manager.go:414] "containerMap: removing container" podUID="793c02dd-468b-4308-bac2-67d04e40579c" containerName="registry" Nov 03 20:52:23.620407 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.620362 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="793c02dd-468b-4308-bac2-67d04e40579c" containerName="registry" Nov 03 20:52:23.620407 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.620381 4963 cpu_manager.go:414] "containerMap: removing container" podUID="15d37b2c-d051-419e-b0cb-d761cac6ff74" containerName="registry" Nov 03 20:52:23.620407 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.620386 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15d37b2c-d051-419e-b0cb-d761cac6ff74" containerName="registry" Nov 03 20:52:23.620407 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.620399 4963 cpu_manager.go:414] "containerMap: removing container" podUID="1f33677f-ee87-4cd1-bdb0-b74000834db2" containerName="insights-gathering" Nov 03 20:52:23.620407 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.620404 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f33677f-ee87-4cd1-bdb0-b74000834db2" containerName="insights-gathering" Nov 03 20:52:23.620556 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.620463 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="793c02dd-468b-4308-bac2-67d04e40579c" containerName="registry" Nov 03 20:52:23.620556 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.620472 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="15d37b2c-d051-419e-b0cb-d761cac6ff74" containerName="registry" Nov 03 20:52:23.620556 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.620479 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f33677f-ee87-4cd1-bdb0-b74000834db2" containerName="insights-gathering" Nov 03 20:52:23.622602 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.622583 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.672797 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.672708 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 20:52:23.706620 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.706585 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec5bfd4-81b3-4a81-a2be-9d832439813f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.706798 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.706632 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ec5bfd4-81b3-4a81-a2be-9d832439813f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.706798 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.706655 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq67k\" (UniqueName: \"kubernetes.io/projected/6ec5bfd4-81b3-4a81-a2be-9d832439813f-kube-api-access-nq67k\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.706798 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.706780 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ec5bfd4-81b3-4a81-a2be-9d832439813f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.706933 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.706817 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ec5bfd4-81b3-4a81-a2be-9d832439813f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.706933 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.706852 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ec5bfd4-81b3-4a81-a2be-9d832439813f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.707015 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.706954 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec5bfd4-81b3-4a81-a2be-9d832439813f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.707051 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.707035 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ec5bfd4-81b3-4a81-a2be-9d832439813f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.707118 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.707099 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec5bfd4-81b3-4a81-a2be-9d832439813f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.715194 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.715167 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v"] Nov 03 20:52:23.734637 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.730720 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-nn44p\"" Nov 03 20:52:23.796856 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.796821 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:23.807738 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.807711 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ec5bfd4-81b3-4a81-a2be-9d832439813f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.807887 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.807743 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ec5bfd4-81b3-4a81-a2be-9d832439813f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.807887 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.807805 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ec5bfd4-81b3-4a81-a2be-9d832439813f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.807887 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.807836 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec5bfd4-81b3-4a81-a2be-9d832439813f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.807887 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.807854 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ec5bfd4-81b3-4a81-a2be-9d832439813f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.807887 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.807882 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec5bfd4-81b3-4a81-a2be-9d832439813f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.808142 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.807940 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec5bfd4-81b3-4a81-a2be-9d832439813f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.808142 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.808057 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ec5bfd4-81b3-4a81-a2be-9d832439813f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.808142 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.808102 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq67k\" (UniqueName: \"kubernetes.io/projected/6ec5bfd4-81b3-4a81-a2be-9d832439813f-kube-api-access-nq67k\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.808355 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.808331 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec5bfd4-81b3-4a81-a2be-9d832439813f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.808437 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.808353 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ec5bfd4-81b3-4a81-a2be-9d832439813f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.808437 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.808411 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec5bfd4-81b3-4a81-a2be-9d832439813f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.808564 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.808544 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec5bfd4-81b3-4a81-a2be-9d832439813f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.808788 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.808769 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ec5bfd4-81b3-4a81-a2be-9d832439813f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.810660 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.810642 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ec5bfd4-81b3-4a81-a2be-9d832439813f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.810888 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.810869 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ec5bfd4-81b3-4a81-a2be-9d832439813f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.866130 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.866100 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ec5bfd4-81b3-4a81-a2be-9d832439813f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.866268 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.866098 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq67k\" (UniqueName: \"kubernetes.io/projected/6ec5bfd4-81b3-4a81-a2be-9d832439813f-kube-api-access-nq67k\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v\" (UID: \"6ec5bfd4-81b3-4a81-a2be-9d832439813f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:23.934558 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:23.934470 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:24.034203 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:24.034169 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:24.358973 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:24.358935 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v"] Nov 03 20:52:24.365892 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:52:24.365863 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec5bfd4_81b3_4a81_a2be_9d832439813f.slice/crio-46a75beded2f3f634999e0ab1c0db3df1ae9bc4b21aba75d60867d2047f6cf97 WatchSource:0}: Error finding container 46a75beded2f3f634999e0ab1c0db3df1ae9bc4b21aba75d60867d2047f6cf97: Status 404 returned error can't find the container with id 46a75beded2f3f634999e0ab1c0db3df1ae9bc4b21aba75d60867d2047f6cf97 Nov 03 20:52:25.310682 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:25.310645 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" event={"ID":"6ec5bfd4-81b3-4a81-a2be-9d832439813f","Type":"ContainerStarted","Data":"46a75beded2f3f634999e0ab1c0db3df1ae9bc4b21aba75d60867d2047f6cf97"} Nov 03 20:52:26.207483 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.207445 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d"] Nov 03 20:52:26.211319 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.211295 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d" Nov 03 20:52:26.238943 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.238914 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5"] Nov 03 20:52:26.241230 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.241211 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" Nov 03 20:52:26.261252 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.261229 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 20:52:26.283690 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.283660 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 20:52:26.283858 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.283727 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-v424j\"" Nov 03 20:52:26.309939 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.309889 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d"] Nov 03 20:52:26.310879 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.310856 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5"] Nov 03 20:52:26.330174 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.330139 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d\" (UID: \"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d" Nov 03 20:52:26.330309 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.330185 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-478fx\" (UniqueName: \"kubernetes.io/projected/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-kube-api-access-478fx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d\" (UID: \"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d" Nov 03 20:52:26.330309 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.330267 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpmm6\" (UniqueName: \"kubernetes.io/projected/9e98e374-909f-47db-9f46-d4dc56748e41-kube-api-access-zpmm6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5\" (UID: \"9e98e374-909f-47db-9f46-d4dc56748e41\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" Nov 03 20:52:26.330444 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.330339 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d\" (UID: \"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d" Nov 03 20:52:26.330444 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.330379 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e98e374-909f-47db-9f46-d4dc56748e41-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5\" (UID: \"9e98e374-909f-47db-9f46-d4dc56748e41\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" Nov 03 20:52:26.330444 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.330402 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e98e374-909f-47db-9f46-d4dc56748e41-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5\" (UID: \"9e98e374-909f-47db-9f46-d4dc56748e41\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" Nov 03 20:52:26.362004 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.361972 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-pmxkh\"" Nov 03 20:52:26.430790 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.430750 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d\" (UID: \"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d" Nov 03 20:52:26.430790 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.430794 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-478fx\" (UniqueName: \"kubernetes.io/projected/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-kube-api-access-478fx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d\" (UID: \"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d" Nov 03 20:52:26.431038 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.430826 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpmm6\" (UniqueName: \"kubernetes.io/projected/9e98e374-909f-47db-9f46-d4dc56748e41-kube-api-access-zpmm6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5\" (UID: \"9e98e374-909f-47db-9f46-d4dc56748e41\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" Nov 03 20:52:26.431038 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.430844 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d\" (UID: \"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d" Nov 03 20:52:26.431038 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.430869 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e98e374-909f-47db-9f46-d4dc56748e41-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5\" (UID: \"9e98e374-909f-47db-9f46-d4dc56748e41\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" Nov 03 20:52:26.431038 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.430893 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e98e374-909f-47db-9f46-d4dc56748e41-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5\" (UID: \"9e98e374-909f-47db-9f46-d4dc56748e41\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" Nov 03 20:52:26.431221 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.431200 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d\" (UID: \"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d" Nov 03 20:52:26.431277 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.431245 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d\" (UID: \"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d" Nov 03 20:52:26.431332 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.431299 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e98e374-909f-47db-9f46-d4dc56748e41-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5\" (UID: \"9e98e374-909f-47db-9f46-d4dc56748e41\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" Nov 03 20:52:26.431332 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.431250 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e98e374-909f-47db-9f46-d4dc56748e41-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5\" (UID: \"9e98e374-909f-47db-9f46-d4dc56748e41\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" Nov 03 20:52:26.489971 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.489876 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-478fx\" (UniqueName: \"kubernetes.io/projected/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-kube-api-access-478fx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d\" (UID: \"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d" Nov 03 20:52:26.489971 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.489876 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpmm6\" (UniqueName: \"kubernetes.io/projected/9e98e374-909f-47db-9f46-d4dc56748e41-kube-api-access-zpmm6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5\" (UID: \"9e98e374-909f-47db-9f46-d4dc56748e41\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" Nov 03 20:52:26.525049 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.525014 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d" Nov 03 20:52:26.554061 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.554034 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" Nov 03 20:52:26.954459 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.954425 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d"] Nov 03 20:52:26.957645 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:26.957619 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5"] Nov 03 20:52:26.961509 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:52:26.961484 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ccfd1e8_9e48_45cf_abbd_7b04c88bcf41.slice/crio-7c6e3a618046b5e2f2857916a31f6d019533bda9fa92dd6298fbcce0a24a7813 WatchSource:0}: Error finding container 7c6e3a618046b5e2f2857916a31f6d019533bda9fa92dd6298fbcce0a24a7813: Status 404 returned error can't find the container with id 7c6e3a618046b5e2f2857916a31f6d019533bda9fa92dd6298fbcce0a24a7813 Nov 03 20:52:27.001823 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:52:27.001786 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e98e374_909f_47db_9f46_d4dc56748e41.slice/crio-e2be7a1b7d8c41050c9fb8b8ed2c1c37e6f0efd61614fc5dd92fcb90540e6539 WatchSource:0}: Error finding container e2be7a1b7d8c41050c9fb8b8ed2c1c37e6f0efd61614fc5dd92fcb90540e6539: Status 404 returned error can't find the container with id e2be7a1b7d8c41050c9fb8b8ed2c1c37e6f0efd61614fc5dd92fcb90540e6539 Nov 03 20:52:27.319791 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:27.319702 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" event={"ID":"9e98e374-909f-47db-9f46-d4dc56748e41","Type":"ContainerStarted","Data":"e2be7a1b7d8c41050c9fb8b8ed2c1c37e6f0efd61614fc5dd92fcb90540e6539"} Nov 03 20:52:27.320731 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:27.320703 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d" event={"ID":"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41","Type":"ContainerStarted","Data":"7c6e3a618046b5e2f2857916a31f6d019533bda9fa92dd6298fbcce0a24a7813"} Nov 03 20:52:27.676549 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:27.676513 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:30.972449 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:30.972416 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 20:52:30.973022 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:30.972953 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 20:52:33.045575 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:33.045540 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:52:33.045930 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:33.045637 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:52:33.045930 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:33.045690 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:52:33.353489 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:33.353399 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" event={"ID":"6ec5bfd4-81b3-4a81-a2be-9d832439813f","Type":"ContainerStarted","Data":"22bce345fbd2c2b2f7de8b7ec0f2efee88b738d8667705b9b3fcd9d671b57f33"} Nov 03 20:52:33.470212 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:33.470154 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" podStartSLOduration=1.793360455 podStartE2EDuration="10.47013986s" totalImagesPullingTime="8.676779405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:33.469078423 +0000 UTC m=+302.629150687" watchObservedRunningTime="2026-11-03 20:52:33.47013986 +0000 UTC m=+302.630212131" Nov 03 20:52:33.935663 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:33.935579 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:34.359340 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:34.359303 4963 generic.go:356] "Generic (PLEG): container finished" podID="9e98e374-909f-47db-9f46-d4dc56748e41" containerID="c6d5dfd4dc4decbe94fa388eeda84067b83c320ab5809301a8055ef6d594eb46" exitCode=0 Nov 03 20:52:34.359824 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:34.359390 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" event={"ID":"9e98e374-909f-47db-9f46-d4dc56748e41","Type":"ContainerDied","Data":"c6d5dfd4dc4decbe94fa388eeda84067b83c320ab5809301a8055ef6d594eb46"} Nov 03 20:52:34.360695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:34.360678 4963 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:52:34.939517 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:34.939485 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:35.364998 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:35.364960 4963 generic.go:356] "Generic (PLEG): container finished" podID="6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41" containerID="4b88faee2e134657472e132dbab3625fea38c3c9f3f676805b7d79f4cc2cd776" exitCode=0 Nov 03 20:52:35.365406 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:35.365048 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d" event={"ID":"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41","Type":"ContainerDied","Data":"4b88faee2e134657472e132dbab3625fea38c3c9f3f676805b7d79f4cc2cd776"} Nov 03 20:52:35.365448 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:35.365432 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:35.366452 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:35.366432 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v" Nov 03 20:52:37.375960 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:37.375897 4963 generic.go:356] "Generic (PLEG): container finished" podID="9e98e374-909f-47db-9f46-d4dc56748e41" containerID="2a66eac958fed0aaf62e690bb24f5cb7e4a97663630f7d6bb2197807e00f5457" exitCode=0 Nov 03 20:52:37.376414 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:37.375997 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" event={"ID":"9e98e374-909f-47db-9f46-d4dc56748e41","Type":"ContainerDied","Data":"2a66eac958fed0aaf62e690bb24f5cb7e4a97663630f7d6bb2197807e00f5457"} Nov 03 20:52:42.532234 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:52:42.532139 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:52:43.403053 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:43.403018 4963 generic.go:356] "Generic (PLEG): container finished" podID="6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41" containerID="d5333f2ccc99970614e9b12402e4394940f8147bfb52e6a882f54773a6c561c9" exitCode=0 Nov 03 20:52:43.403241 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:43.403087 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d" event={"ID":"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41","Type":"ContainerDied","Data":"d5333f2ccc99970614e9b12402e4394940f8147bfb52e6a882f54773a6c561c9"} Nov 03 20:52:47.425439 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:47.425393 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" event={"ID":"9e98e374-909f-47db-9f46-d4dc56748e41","Type":"ContainerStarted","Data":"d693de0faac378ad29327226fef4687c4518a556cc64840ea2d6c0b2fbe466c7"} Nov 03 20:52:47.537678 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:47.537619 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" podStartSLOduration=3.3195705220000002 podStartE2EDuration="21.537600371s" totalImagesPullingTime="18.218029849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:47.537562025 +0000 UTC m=+316.697634300" watchObservedRunningTime="2026-11-03 20:52:47.537600371 +0000 UTC m=+316.697672644" Nov 03 20:52:48.432063 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:48.432028 4963 generic.go:356] "Generic (PLEG): container finished" podID="9e98e374-909f-47db-9f46-d4dc56748e41" containerID="d693de0faac378ad29327226fef4687c4518a556cc64840ea2d6c0b2fbe466c7" exitCode=0 Nov 03 20:52:48.432540 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:48.432110 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" event={"ID":"9e98e374-909f-47db-9f46-d4dc56748e41","Type":"ContainerDied","Data":"d693de0faac378ad29327226fef4687c4518a556cc64840ea2d6c0b2fbe466c7"} Nov 03 20:52:49.563801 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:49.563772 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" Nov 03 20:52:49.750462 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:49.750420 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e98e374-909f-47db-9f46-d4dc56748e41-util\" (UniqueName: \"kubernetes.io/empty-dir/9e98e374-909f-47db-9f46-d4dc56748e41-util\") pod \"9e98e374-909f-47db-9f46-d4dc56748e41\" (UID: \"9e98e374-909f-47db-9f46-d4dc56748e41\") " Nov 03 20:52:49.750462 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:49.750464 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e98e374-909f-47db-9f46-d4dc56748e41-kube-api-access-zpmm6\" (UniqueName: \"kubernetes.io/projected/9e98e374-909f-47db-9f46-d4dc56748e41-kube-api-access-zpmm6\") pod \"9e98e374-909f-47db-9f46-d4dc56748e41\" (UID: \"9e98e374-909f-47db-9f46-d4dc56748e41\") " Nov 03 20:52:49.750706 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:49.750517 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e98e374-909f-47db-9f46-d4dc56748e41-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e98e374-909f-47db-9f46-d4dc56748e41-bundle\") pod \"9e98e374-909f-47db-9f46-d4dc56748e41\" (UID: \"9e98e374-909f-47db-9f46-d4dc56748e41\") " Nov 03 20:52:49.751122 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:49.751095 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e98e374-909f-47db-9f46-d4dc56748e41-bundle" pod "9e98e374-909f-47db-9f46-d4dc56748e41" (UID: "9e98e374-909f-47db-9f46-d4dc56748e41"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:49.752742 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:49.752712 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e98e374-909f-47db-9f46-d4dc56748e41-kube-api-access-zpmm6" pod "9e98e374-909f-47db-9f46-d4dc56748e41" (UID: "9e98e374-909f-47db-9f46-d4dc56748e41"). InnerVolumeSpecName "kube-api-access-zpmm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:49.754557 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:49.754529 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e98e374-909f-47db-9f46-d4dc56748e41-util" pod "9e98e374-909f-47db-9f46-d4dc56748e41" (UID: "9e98e374-909f-47db-9f46-d4dc56748e41"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:49.851891 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:49.851831 4963 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e98e374-909f-47db-9f46-d4dc56748e41-util\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:49.851891 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:49.851886 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpmm6\" (UniqueName: \"kubernetes.io/projected/9e98e374-909f-47db-9f46-d4dc56748e41-kube-api-access-zpmm6\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:49.852176 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:49.851899 4963 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e98e374-909f-47db-9f46-d4dc56748e41-bundle\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:50.443285 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:50.443250 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" Nov 03 20:52:50.443455 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:50.443277 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch56j5" event={"ID":"9e98e374-909f-47db-9f46-d4dc56748e41","Type":"ContainerDied","Data":"e2be7a1b7d8c41050c9fb8b8ed2c1c37e6f0efd61614fc5dd92fcb90540e6539"} Nov 03 20:52:50.443455 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:50.443339 4963 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2be7a1b7d8c41050c9fb8b8ed2c1c37e6f0efd61614fc5dd92fcb90540e6539" Nov 03 20:52:53.458021 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:53.457986 4963 generic.go:356] "Generic (PLEG): container finished" podID="6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41" containerID="096e5a8220bea59223fb1972ed79a1416f16c064edc02537c0a24dd1368fb835" exitCode=0 Nov 03 20:52:53.458499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:53.458041 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d" event={"ID":"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41","Type":"ContainerDied","Data":"096e5a8220bea59223fb1972ed79a1416f16c064edc02537c0a24dd1368fb835"} Nov 03 20:52:54.588003 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:54.587975 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d" Nov 03 20:52:54.690451 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:54.690411 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-util\" (UniqueName: \"kubernetes.io/empty-dir/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-util\") pod \"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41\" (UID: \"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41\") " Nov 03 20:52:54.690655 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:54.690461 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-kube-api-access-478fx\" (UniqueName: \"kubernetes.io/projected/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-kube-api-access-478fx\") pod \"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41\" (UID: \"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41\") " Nov 03 20:52:54.690655 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:54.690494 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-bundle\") pod \"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41\" (UID: \"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41\") " Nov 03 20:52:54.691014 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:54.690991 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-bundle" pod "6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41" (UID: "6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:54.692632 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:54.692608 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-kube-api-access-478fx" pod "6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41" (UID: "6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41"). InnerVolumeSpecName "kube-api-access-478fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:54.695218 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:54.695194 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-util" pod "6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41" (UID: "6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:54.792139 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:54.792052 4963 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-util\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:54.792139 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:54.792084 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-478fx\" (UniqueName: \"kubernetes.io/projected/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-kube-api-access-478fx\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:54.792139 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:54.792093 4963 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41-bundle\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:55.468222 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:55.468190 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d" event={"ID":"6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41","Type":"ContainerDied","Data":"7c6e3a618046b5e2f2857916a31f6d019533bda9fa92dd6298fbcce0a24a7813"} Nov 03 20:52:55.468390 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:55.468236 4963 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c6e3a618046b5e2f2857916a31f6d019533bda9fa92dd6298fbcce0a24a7813" Nov 03 20:52:55.468390 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:55.468278 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jn76d" Nov 03 20:52:56.658117 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:56.658084 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:56.849403 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:52:56.849371 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:05.307981 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:05.307948 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:06.322152 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:06.322117 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:08.331233 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:08.331202 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:08.512867 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:08.512832 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:09.727295 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.727256 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:09.901394 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.901353 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq"] Nov 03 20:53:09.901691 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.901679 4963 cpu_manager.go:414] "containerMap: removing container" podUID="6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41" containerName="util" Nov 03 20:53:09.901737 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.901691 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41" containerName="util" Nov 03 20:53:09.901737 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.901706 4963 cpu_manager.go:414] "containerMap: removing container" podUID="9e98e374-909f-47db-9f46-d4dc56748e41" containerName="extract" Nov 03 20:53:09.901737 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.901711 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e98e374-909f-47db-9f46-d4dc56748e41" containerName="extract" Nov 03 20:53:09.901737 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.901717 4963 cpu_manager.go:414] "containerMap: removing container" podUID="6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41" containerName="extract" Nov 03 20:53:09.901737 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.901722 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41" containerName="extract" Nov 03 20:53:09.901737 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.901735 4963 cpu_manager.go:414] "containerMap: removing container" podUID="9e98e374-909f-47db-9f46-d4dc56748e41" containerName="pull" Nov 03 20:53:09.901937 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.901740 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e98e374-909f-47db-9f46-d4dc56748e41" containerName="pull" Nov 03 20:53:09.901937 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.901748 4963 cpu_manager.go:414] "containerMap: removing container" podUID="9e98e374-909f-47db-9f46-d4dc56748e41" containerName="util" Nov 03 20:53:09.901937 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.901752 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e98e374-909f-47db-9f46-d4dc56748e41" containerName="util" Nov 03 20:53:09.901937 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.901763 4963 cpu_manager.go:414] "containerMap: removing container" podUID="6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41" containerName="pull" Nov 03 20:53:09.901937 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.901768 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41" containerName="pull" Nov 03 20:53:09.901937 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.901818 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e98e374-909f-47db-9f46-d4dc56748e41" containerName="extract" Nov 03 20:53:09.901937 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.901826 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ccfd1e8-9e48-45cf-abbd-7b04c88bcf41" containerName="extract" Nov 03 20:53:09.903963 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.903944 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:09.954556 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.954528 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 20:53:09.957075 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.957058 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 20:53:09.957168 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:09.957144 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 20:53:10.006625 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.006546 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq"] Nov 03 20:53:10.020477 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.020438 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/202a76c0-233d-4f6f-84fa-a2574fce92d8-manifests\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qmccq\" (UID: \"202a76c0-233d-4f6f-84fa-a2574fce92d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.020657 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.020528 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbfz5\" (UniqueName: \"kubernetes.io/projected/202a76c0-233d-4f6f-84fa-a2574fce92d8-kube-api-access-pbfz5\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qmccq\" (UID: \"202a76c0-233d-4f6f-84fa-a2574fce92d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.020657 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.020550 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/202a76c0-233d-4f6f-84fa-a2574fce92d8-tmp\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qmccq\" (UID: \"202a76c0-233d-4f6f-84fa-a2574fce92d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.020782 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.020731 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/202a76c0-233d-4f6f-84fa-a2574fce92d8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qmccq\" (UID: \"202a76c0-233d-4f6f-84fa-a2574fce92d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.020851 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.020803 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/202a76c0-233d-4f6f-84fa-a2574fce92d8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qmccq\" (UID: \"202a76c0-233d-4f6f-84fa-a2574fce92d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.020851 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.020839 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/202a76c0-233d-4f6f-84fa-a2574fce92d8-metrics-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qmccq\" (UID: \"202a76c0-233d-4f6f-84fa-a2574fce92d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.057188 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.057159 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 20:53:10.057371 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.057224 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-pc7cb\"" Nov 03 20:53:10.057670 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.057656 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 20:53:10.064594 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.064559 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:10.121472 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.121433 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/202a76c0-233d-4f6f-84fa-a2574fce92d8-manifests\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qmccq\" (UID: \"202a76c0-233d-4f6f-84fa-a2574fce92d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.121677 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.121489 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbfz5\" (UniqueName: \"kubernetes.io/projected/202a76c0-233d-4f6f-84fa-a2574fce92d8-kube-api-access-pbfz5\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qmccq\" (UID: \"202a76c0-233d-4f6f-84fa-a2574fce92d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.121677 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.121510 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/202a76c0-233d-4f6f-84fa-a2574fce92d8-tmp\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qmccq\" (UID: \"202a76c0-233d-4f6f-84fa-a2574fce92d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.121677 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.121534 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/202a76c0-233d-4f6f-84fa-a2574fce92d8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qmccq\" (UID: \"202a76c0-233d-4f6f-84fa-a2574fce92d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.121677 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.121576 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/202a76c0-233d-4f6f-84fa-a2574fce92d8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qmccq\" (UID: \"202a76c0-233d-4f6f-84fa-a2574fce92d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.121677 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.121600 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/202a76c0-233d-4f6f-84fa-a2574fce92d8-metrics-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qmccq\" (UID: \"202a76c0-233d-4f6f-84fa-a2574fce92d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.122004 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.121981 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/202a76c0-233d-4f6f-84fa-a2574fce92d8-manifests\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qmccq\" (UID: \"202a76c0-233d-4f6f-84fa-a2574fce92d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.123979 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.123947 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/202a76c0-233d-4f6f-84fa-a2574fce92d8-tmp\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qmccq\" (UID: \"202a76c0-233d-4f6f-84fa-a2574fce92d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.124287 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.124266 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/202a76c0-233d-4f6f-84fa-a2574fce92d8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qmccq\" (UID: \"202a76c0-233d-4f6f-84fa-a2574fce92d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.124353 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.124298 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/202a76c0-233d-4f6f-84fa-a2574fce92d8-metrics-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qmccq\" (UID: \"202a76c0-233d-4f6f-84fa-a2574fce92d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.124409 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.124354 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/202a76c0-233d-4f6f-84fa-a2574fce92d8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qmccq\" (UID: \"202a76c0-233d-4f6f-84fa-a2574fce92d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.180540 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.180503 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbfz5\" (UniqueName: \"kubernetes.io/projected/202a76c0-233d-4f6f-84fa-a2574fce92d8-kube-api-access-pbfz5\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qmccq\" (UID: \"202a76c0-233d-4f6f-84fa-a2574fce92d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.216466 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.216409 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:10.256844 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.256765 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:10.444740 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.444696 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:10.706689 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.706651 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq"] Nov 03 20:53:10.711986 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:53:10.711954 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202a76c0_233d_4f6f_84fa_a2574fce92d8.slice/crio-4fb0953853e747108400b543c7409e5c6bd5de7cbcfc9accf4a2e54342a41d40 WatchSource:0}: Error finding container 4fb0953853e747108400b543c7409e5c6bd5de7cbcfc9accf4a2e54342a41d40: Status 404 returned error can't find the container with id 4fb0953853e747108400b543c7409e5c6bd5de7cbcfc9accf4a2e54342a41d40 Nov 03 20:53:10.884572 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.884541 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-mcs6w"] Nov 03 20:53:10.887174 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.887154 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mcs6w" Nov 03 20:53:10.928696 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.928658 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjt98\" (UniqueName: \"kubernetes.io/projected/3a26aaaf-9eac-483f-b59e-2fdd595e214c-kube-api-access-bjt98\") pod \"authorino-operator-7b7bb4bc77-mcs6w\" (UID: \"3a26aaaf-9eac-483f-b59e-2fdd595e214c\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-mcs6w" Nov 03 20:53:10.936969 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.936940 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 20:53:10.937142 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.937125 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-f8sww\"" Nov 03 20:53:10.937142 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.937126 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 20:53:10.946337 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:10.946305 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-mcs6w"] Nov 03 20:53:11.029793 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:11.029706 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjt98\" (UniqueName: \"kubernetes.io/projected/3a26aaaf-9eac-483f-b59e-2fdd595e214c-kube-api-access-bjt98\") pod \"authorino-operator-7b7bb4bc77-mcs6w\" (UID: \"3a26aaaf-9eac-483f-b59e-2fdd595e214c\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-mcs6w" Nov 03 20:53:11.088223 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:11.088193 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjt98\" (UniqueName: \"kubernetes.io/projected/3a26aaaf-9eac-483f-b59e-2fdd595e214c-kube-api-access-bjt98\") pod \"authorino-operator-7b7bb4bc77-mcs6w\" (UID: \"3a26aaaf-9eac-483f-b59e-2fdd595e214c\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-mcs6w" Nov 03 20:53:11.199105 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:11.199066 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mcs6w" Nov 03 20:53:11.535515 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:11.535473 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" event={"ID":"202a76c0-233d-4f6f-84fa-a2574fce92d8","Type":"ContainerStarted","Data":"4fb0953853e747108400b543c7409e5c6bd5de7cbcfc9accf4a2e54342a41d40"} Nov 03 20:53:11.587397 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:11.587357 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-mcs6w"] Nov 03 20:53:11.592402 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:53:11.592375 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a26aaaf_9eac_483f_b59e_2fdd595e214c.slice/crio-720f1a29608bbda1be86c944261c684989ff4f59e1039d6b58d45ce7d3e51181 WatchSource:0}: Error finding container 720f1a29608bbda1be86c944261c684989ff4f59e1039d6b58d45ce7d3e51181: Status 404 returned error can't find the container with id 720f1a29608bbda1be86c944261c684989ff4f59e1039d6b58d45ce7d3e51181 Nov 03 20:53:12.124915 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:12.124879 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:12.534155 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:53:12.534120 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:53:12.541999 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:12.541950 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mcs6w" event={"ID":"3a26aaaf-9eac-483f-b59e-2fdd595e214c","Type":"ContainerStarted","Data":"720f1a29608bbda1be86c944261c684989ff4f59e1039d6b58d45ce7d3e51181"} Nov 03 20:53:13.944452 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:13.944412 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:14.552607 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:14.552564 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" event={"ID":"202a76c0-233d-4f6f-84fa-a2574fce92d8","Type":"ContainerStarted","Data":"ba1abdf122515428ddd6662c53d412ed3d37dc54cb5166220a3082bf49d29775"} Nov 03 20:53:14.843391 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:14.843353 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:15.033668 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:15.033630 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:15.368646 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:15.368561 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:15.559830 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:15.559789 4963 generic.go:356] "Generic (PLEG): container finished" podID="202a76c0-233d-4f6f-84fa-a2574fce92d8" containerID="ba1abdf122515428ddd6662c53d412ed3d37dc54cb5166220a3082bf49d29775" exitCode=0 Nov 03 20:53:15.560042 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:15.559865 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" event={"ID":"202a76c0-233d-4f6f-84fa-a2574fce92d8","Type":"ContainerDied","Data":"ba1abdf122515428ddd6662c53d412ed3d37dc54cb5166220a3082bf49d29775"} Nov 03 20:53:15.561768 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:15.561751 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:15.915553 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:15.915519 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:17.570684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:17.570591 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" event={"ID":"202a76c0-233d-4f6f-84fa-a2574fce92d8","Type":"ContainerStarted","Data":"00430a75df169bc5375568bf47727c127101cb67d0e6f65a908e9c431da5a77a"} Nov 03 20:53:17.571138 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:17.570735 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:17.572288 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:17.572262 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mcs6w" event={"ID":"3a26aaaf-9eac-483f-b59e-2fdd595e214c","Type":"ContainerStarted","Data":"5994785039ba527a0a70d4bb391f8dc62dba599f7b847afcc7386553bcd551ab"} Nov 03 20:53:17.572404 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:17.572336 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mcs6w" Nov 03 20:53:17.846102 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:17.846008 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" podStartSLOduration=3.509008951 podStartE2EDuration="8.845993164s" totalImagesPullingTime="4.751513085s" totalInitContainerRuntime="585.471128ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:17.79707308 +0000 UTC m=+346.957145352" watchObservedRunningTime="2026-11-03 20:53:17.845993164 +0000 UTC m=+347.006065435" Nov 03 20:53:17.910791 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:17.910718 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mcs6w" podStartSLOduration=2.294002781 podStartE2EDuration="7.910699432s" totalImagesPullingTime="5.616696651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:17.909610353 +0000 UTC m=+347.069682627" watchObservedRunningTime="2026-11-03 20:53:17.910699432 +0000 UTC m=+347.070771704" Nov 03 20:53:17.931490 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:17.931450 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:18.191383 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:18.191284 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:18.975151 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:18.975116 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-lcfpk"] Nov 03 20:53:18.977630 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:18.977605 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lcfpk" Nov 03 20:53:19.034023 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:19.033986 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-hs7zh\"" Nov 03 20:53:19.035707 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:19.035683 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 20:53:19.083078 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:19.083047 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-lcfpk"] Nov 03 20:53:19.105749 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:19.105715 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s96t\" (UniqueName: \"kubernetes.io/projected/5359b257-aafe-4a02-b987-786f2b8e3efc-kube-api-access-7s96t\") pod \"dns-operator-controller-manager-8557cbdb84-lcfpk\" (UID: \"5359b257-aafe-4a02-b987-786f2b8e3efc\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lcfpk" Nov 03 20:53:19.206265 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:19.206213 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s96t\" (UniqueName: \"kubernetes.io/projected/5359b257-aafe-4a02-b987-786f2b8e3efc-kube-api-access-7s96t\") pod \"dns-operator-controller-manager-8557cbdb84-lcfpk\" (UID: \"5359b257-aafe-4a02-b987-786f2b8e3efc\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lcfpk" Nov 03 20:53:19.265340 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:19.265271 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s96t\" (UniqueName: \"kubernetes.io/projected/5359b257-aafe-4a02-b987-786f2b8e3efc-kube-api-access-7s96t\") pod \"dns-operator-controller-manager-8557cbdb84-lcfpk\" (UID: \"5359b257-aafe-4a02-b987-786f2b8e3efc\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lcfpk" Nov 03 20:53:19.294116 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:19.294078 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lcfpk" Nov 03 20:53:19.511859 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:19.511825 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:19.669717 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:19.669633 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-lcfpk"] Nov 03 20:53:19.675753 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:53:19.675713 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5359b257_aafe_4a02_b987_786f2b8e3efc.slice/crio-d68c8e6c4ac780e8a1ae2509275a1f778f897c9821c81cfde0bd5c32daa3c11f WatchSource:0}: Error finding container d68c8e6c4ac780e8a1ae2509275a1f778f897c9821c81cfde0bd5c32daa3c11f: Status 404 returned error can't find the container with id d68c8e6c4ac780e8a1ae2509275a1f778f897c9821c81cfde0bd5c32daa3c11f Nov 03 20:53:20.320121 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:20.320085 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:20.547819 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:20.547782 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:20.588168 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:20.588083 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lcfpk" event={"ID":"5359b257-aafe-4a02-b987-786f2b8e3efc","Type":"ContainerStarted","Data":"d68c8e6c4ac780e8a1ae2509275a1f778f897c9821c81cfde0bd5c32daa3c11f"} Nov 03 20:53:21.122508 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:21.122474 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:22.278158 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:22.278120 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:22.474581 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:22.474540 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:22.587943 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:22.587844 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7"] Nov 03 20:53:22.590437 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:22.590401 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7" Nov 03 20:53:22.639889 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:22.639839 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7"] Nov 03 20:53:22.684943 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:22.684897 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:53:22.685125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:22.684955 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 20:53:22.685125 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:22.684991 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-pdthg\"" Nov 03 20:53:22.736026 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:22.735989 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j9t5\" (UniqueName: \"kubernetes.io/projected/1803099f-37b1-4730-a7e1-d7488a22dd66-kube-api-access-6j9t5\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7\" (UID: \"1803099f-37b1-4730-a7e1-d7488a22dd66\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7" Nov 03 20:53:22.736211 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:22.736094 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1803099f-37b1-4730-a7e1-d7488a22dd66-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7\" (UID: \"1803099f-37b1-4730-a7e1-d7488a22dd66\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7" Nov 03 20:53:22.837527 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:22.837489 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j9t5\" (UniqueName: \"kubernetes.io/projected/1803099f-37b1-4730-a7e1-d7488a22dd66-kube-api-access-6j9t5\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7\" (UID: \"1803099f-37b1-4730-a7e1-d7488a22dd66\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7" Nov 03 20:53:22.837713 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:22.837547 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1803099f-37b1-4730-a7e1-d7488a22dd66-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7\" (UID: \"1803099f-37b1-4730-a7e1-d7488a22dd66\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7" Nov 03 20:53:22.838071 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:22.838018 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1803099f-37b1-4730-a7e1-d7488a22dd66-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7\" (UID: \"1803099f-37b1-4730-a7e1-d7488a22dd66\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7" Nov 03 20:53:22.902486 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:22.902440 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j9t5\" (UniqueName: \"kubernetes.io/projected/1803099f-37b1-4730-a7e1-d7488a22dd66-kube-api-access-6j9t5\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7\" (UID: \"1803099f-37b1-4730-a7e1-d7488a22dd66\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7" Nov 03 20:53:22.905096 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:22.905063 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7" Nov 03 20:53:22.964965 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:22.964938 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:23.332761 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:23.332724 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7"] Nov 03 20:53:23.340708 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:53:23.340677 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1803099f_37b1_4730_a7e1_d7488a22dd66.slice/crio-d3d0e50c5f431fcf8ed6784e460fc2cb0d2fba9544cb462115a7cebe4d14c629 WatchSource:0}: Error finding container d3d0e50c5f431fcf8ed6784e460fc2cb0d2fba9544cb462115a7cebe4d14c629: Status 404 returned error can't find the container with id d3d0e50c5f431fcf8ed6784e460fc2cb0d2fba9544cb462115a7cebe4d14c629 Nov 03 20:53:23.603739 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:23.603650 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7" event={"ID":"1803099f-37b1-4730-a7e1-d7488a22dd66","Type":"ContainerStarted","Data":"d3d0e50c5f431fcf8ed6784e460fc2cb0d2fba9544cb462115a7cebe4d14c629"} Nov 03 20:53:23.605304 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:23.605276 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lcfpk" event={"ID":"5359b257-aafe-4a02-b987-786f2b8e3efc","Type":"ContainerStarted","Data":"a2d1b72b279c556d044c1faafe560f68ce66c03016b5a07eb6f9f5a8e1a9e819"} Nov 03 20:53:23.605459 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:23.605399 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lcfpk" Nov 03 20:53:23.615321 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:23.615297 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:23.723726 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:23.723667 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lcfpk" podStartSLOduration=2.316514412 podStartE2EDuration="5.723651653s" totalImagesPullingTime="3.407137241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:23.722768789 +0000 UTC m=+352.882841060" watchObservedRunningTime="2026-11-03 20:53:23.723651653 +0000 UTC m=+352.883723924" Nov 03 20:53:27.628641 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:27.628549 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7" event={"ID":"1803099f-37b1-4730-a7e1-d7488a22dd66","Type":"ContainerStarted","Data":"6443cfb63114f26aa173744ad7ab133b64e6406f80f7c65a591b87069eafc206"} Nov 03 20:53:27.843432 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:27.843367 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7mcj7" podStartSLOduration=1.96973051 podStartE2EDuration="5.843346502s" totalImagesPullingTime="3.873615992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:27.842003474 +0000 UTC m=+357.002075745" watchObservedRunningTime="2026-11-03 20:53:27.843346502 +0000 UTC m=+357.003418774" Nov 03 20:53:28.160831 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:28.160798 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:28.345819 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:28.345785 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:28.578665 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:28.578630 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mcs6w" Nov 03 20:53:28.578848 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:28.578749 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qmccq" Nov 03 20:53:28.874289 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:28.874200 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:29.184817 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:29.184737 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:29.472111 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:29.472079 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:29.660016 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:29.659983 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:29.715475 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:29.715441 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-9xpb2"] Nov 03 20:53:29.719094 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:29.719073 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9xpb2" Nov 03 20:53:29.772032 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:29.771989 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 20:53:29.772032 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:29.772014 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 20:53:29.772290 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:29.772040 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-pfn77\"" Nov 03 20:53:29.774297 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:29.774264 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-9xpb2"] Nov 03 20:53:29.810292 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:29.810254 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps2vq\" (UniqueName: \"kubernetes.io/projected/e092caa7-b842-4f32-a69d-cb1d9fca7865-kube-api-access-ps2vq\") pod \"cert-manager-webhook-5dbbbf75cc-9xpb2\" (UID: \"e092caa7-b842-4f32-a69d-cb1d9fca7865\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9xpb2" Nov 03 20:53:29.810465 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:29.810427 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e092caa7-b842-4f32-a69d-cb1d9fca7865-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-9xpb2\" (UID: \"e092caa7-b842-4f32-a69d-cb1d9fca7865\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9xpb2" Nov 03 20:53:29.911514 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:29.911466 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps2vq\" (UniqueName: \"kubernetes.io/projected/e092caa7-b842-4f32-a69d-cb1d9fca7865-kube-api-access-ps2vq\") pod \"cert-manager-webhook-5dbbbf75cc-9xpb2\" (UID: \"e092caa7-b842-4f32-a69d-cb1d9fca7865\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9xpb2" Nov 03 20:53:29.912010 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:29.911559 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e092caa7-b842-4f32-a69d-cb1d9fca7865-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-9xpb2\" (UID: \"e092caa7-b842-4f32-a69d-cb1d9fca7865\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9xpb2" Nov 03 20:53:29.969796 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:29.969759 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e092caa7-b842-4f32-a69d-cb1d9fca7865-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-9xpb2\" (UID: \"e092caa7-b842-4f32-a69d-cb1d9fca7865\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9xpb2" Nov 03 20:53:29.970042 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:29.970015 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps2vq\" (UniqueName: \"kubernetes.io/projected/e092caa7-b842-4f32-a69d-cb1d9fca7865-kube-api-access-ps2vq\") pod \"cert-manager-webhook-5dbbbf75cc-9xpb2\" (UID: \"e092caa7-b842-4f32-a69d-cb1d9fca7865\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9xpb2" Nov 03 20:53:30.056188 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:30.056101 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9xpb2" Nov 03 20:53:30.428138 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:30.428054 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-9xpb2"] Nov 03 20:53:30.434547 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:53:30.434505 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode092caa7_b842_4f32_a69d_cb1d9fca7865.slice/crio-91ace0ce679551993814040b6f5401d5ba6fc4c017f8ae8b40e77cf67772fe72 WatchSource:0}: Error finding container 91ace0ce679551993814040b6f5401d5ba6fc4c017f8ae8b40e77cf67772fe72: Status 404 returned error can't find the container with id 91ace0ce679551993814040b6f5401d5ba6fc4c017f8ae8b40e77cf67772fe72 Nov 03 20:53:30.643853 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:30.643821 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9xpb2" event={"ID":"e092caa7-b842-4f32-a69d-cb1d9fca7865","Type":"ContainerStarted","Data":"91ace0ce679551993814040b6f5401d5ba6fc4c017f8ae8b40e77cf67772fe72"} Nov 03 20:53:31.002798 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:31.002759 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:31.616096 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:31.616061 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:33.332768 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:33.332717 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:33.516938 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:33.516882 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:34.612060 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:34.612034 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lcfpk" Nov 03 20:53:34.665683 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:34.665641 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9xpb2" event={"ID":"e092caa7-b842-4f32-a69d-cb1d9fca7865","Type":"ContainerStarted","Data":"fbfdf2ae94646c22d2561ba73bb4b8c8d435b7ed415965896bc62e1c189ed874"} Nov 03 20:53:34.665861 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:34.665760 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9xpb2" Nov 03 20:53:34.866408 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:34.866306 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9xpb2" podStartSLOduration=1.7129676150000002 podStartE2EDuration="5.866290758s" totalImagesPullingTime="4.153323143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:34.866042792 +0000 UTC m=+364.026115065" watchObservedRunningTime="2026-11-03 20:53:34.866290758 +0000 UTC m=+364.026363021" Nov 03 20:53:36.325666 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:36.325623 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:36.507828 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:36.507792 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:36.772541 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:36.772509 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:38.427918 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:38.427873 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:38.608388 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:38.608351 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:39.784312 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:39.784272 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:40.679269 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:40.679240 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9xpb2" Nov 03 20:53:40.761094 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:40.761056 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:40.945675 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:40.945638 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:41.336295 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:41.336220 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:41.951469 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:41.951431 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:42.136249 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:42.136219 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:42.527680 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:53:42.527639 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:53:42.882670 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:42.882583 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:42.907356 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:42.907322 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d69f97dbf-99vsk"] Nov 03 20:53:42.911159 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:42.911130 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d69f97dbf-99vsk" Nov 03 20:53:42.924302 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:42.924270 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrwvx\" (UniqueName: \"kubernetes.io/projected/60769ffb-c98f-4d9e-99b8-e13b5f08f0f6-kube-api-access-qrwvx\") pod \"kube-auth-proxy-d69f97dbf-99vsk\" (UID: \"60769ffb-c98f-4d9e-99b8-e13b5f08f0f6\") " pod="openshift-ingress/kube-auth-proxy-d69f97dbf-99vsk" Nov 03 20:53:42.924302 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:42.924306 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60769ffb-c98f-4d9e-99b8-e13b5f08f0f6-tmp\") pod \"kube-auth-proxy-d69f97dbf-99vsk\" (UID: \"60769ffb-c98f-4d9e-99b8-e13b5f08f0f6\") " pod="openshift-ingress/kube-auth-proxy-d69f97dbf-99vsk" Nov 03 20:53:42.924514 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:42.924430 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60769ffb-c98f-4d9e-99b8-e13b5f08f0f6-tls-certs\") pod \"kube-auth-proxy-d69f97dbf-99vsk\" (UID: \"60769ffb-c98f-4d9e-99b8-e13b5f08f0f6\") " pod="openshift-ingress/kube-auth-proxy-d69f97dbf-99vsk" Nov 03 20:53:42.962409 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:42.962366 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 20:53:43.011512 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:43.011477 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d69f97dbf-99vsk"] Nov 03 20:53:43.025511 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:43.025477 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrwvx\" (UniqueName: \"kubernetes.io/projected/60769ffb-c98f-4d9e-99b8-e13b5f08f0f6-kube-api-access-qrwvx\") pod \"kube-auth-proxy-d69f97dbf-99vsk\" (UID: \"60769ffb-c98f-4d9e-99b8-e13b5f08f0f6\") " pod="openshift-ingress/kube-auth-proxy-d69f97dbf-99vsk" Nov 03 20:53:43.025695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:43.025558 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60769ffb-c98f-4d9e-99b8-e13b5f08f0f6-tmp\") pod \"kube-auth-proxy-d69f97dbf-99vsk\" (UID: \"60769ffb-c98f-4d9e-99b8-e13b5f08f0f6\") " pod="openshift-ingress/kube-auth-proxy-d69f97dbf-99vsk" Nov 03 20:53:43.025695 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:43.025634 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60769ffb-c98f-4d9e-99b8-e13b5f08f0f6-tls-certs\") pod \"kube-auth-proxy-d69f97dbf-99vsk\" (UID: \"60769ffb-c98f-4d9e-99b8-e13b5f08f0f6\") " pod="openshift-ingress/kube-auth-proxy-d69f97dbf-99vsk" Nov 03 20:53:43.027961 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:43.027940 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60769ffb-c98f-4d9e-99b8-e13b5f08f0f6-tmp\") pod \"kube-auth-proxy-d69f97dbf-99vsk\" (UID: \"60769ffb-c98f-4d9e-99b8-e13b5f08f0f6\") " pod="openshift-ingress/kube-auth-proxy-d69f97dbf-99vsk" Nov 03 20:53:43.056534 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:43.056502 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 20:53:43.057199 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:43.057175 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-64knl\"" Nov 03 20:53:43.069206 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:43.069174 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60769ffb-c98f-4d9e-99b8-e13b5f08f0f6-tls-certs\") pod \"kube-auth-proxy-d69f97dbf-99vsk\" (UID: \"60769ffb-c98f-4d9e-99b8-e13b5f08f0f6\") " pod="openshift-ingress/kube-auth-proxy-d69f97dbf-99vsk" Nov 03 20:53:43.072444 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:43.072407 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:43.084778 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:43.084742 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrwvx\" (UniqueName: \"kubernetes.io/projected/60769ffb-c98f-4d9e-99b8-e13b5f08f0f6-kube-api-access-qrwvx\") pod \"kube-auth-proxy-d69f97dbf-99vsk\" (UID: \"60769ffb-c98f-4d9e-99b8-e13b5f08f0f6\") " pod="openshift-ingress/kube-auth-proxy-d69f97dbf-99vsk" Nov 03 20:53:43.222939 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:43.222895 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d69f97dbf-99vsk" Nov 03 20:53:43.318454 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:43.318418 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:43.530705 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:43.530611 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:43.595255 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:43.595216 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d69f97dbf-99vsk"] Nov 03 20:53:43.601889 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:53:43.601864 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60769ffb_c98f_4d9e_99b8_e13b5f08f0f6.slice/crio-8568256f5f79673c3e98634ff3f51d581e86ad58e1abfc4234d8d97fef3b8a59 WatchSource:0}: Error finding container 8568256f5f79673c3e98634ff3f51d581e86ad58e1abfc4234d8d97fef3b8a59: Status 404 returned error can't find the container with id 8568256f5f79673c3e98634ff3f51d581e86ad58e1abfc4234d8d97fef3b8a59 Nov 03 20:53:43.714788 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:43.714753 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d69f97dbf-99vsk" event={"ID":"60769ffb-c98f-4d9e-99b8-e13b5f08f0f6","Type":"ContainerStarted","Data":"8568256f5f79673c3e98634ff3f51d581e86ad58e1abfc4234d8d97fef3b8a59"} Nov 03 20:53:43.766043 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:43.765999 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:44.936275 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:44.936185 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:45.118681 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:45.118638 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:45.662562 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:45.662524 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:45.909277 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:45.909243 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:46.093053 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:46.093021 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:46.730947 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:46.730889 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d69f97dbf-99vsk" event={"ID":"60769ffb-c98f-4d9e-99b8-e13b5f08f0f6","Type":"ContainerStarted","Data":"093b90a48475679a3eff98818f8a348f3d1d790d649ac050beebeb661d78b03a"} Nov 03 20:53:46.842442 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:46.842340 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d69f97dbf-99vsk" podStartSLOduration=1.94624884 podStartE2EDuration="4.842323416s" totalImagesPullingTime="2.896074576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:46.841021532 +0000 UTC m=+376.001093815" watchObservedRunningTime="2026-11-03 20:53:46.842323416 +0000 UTC m=+376.002395688" Nov 03 20:53:46.877133 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:46.877101 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dhrl6"] Nov 03 20:53:46.880566 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:46.880543 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-dhrl6" Nov 03 20:53:46.930402 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:46.930367 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dq8fb\"" Nov 03 20:53:46.946411 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:46.946376 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dhrl6"] Nov 03 20:53:46.953514 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:46.953486 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw7fr\" (UniqueName: \"kubernetes.io/projected/c36e2235-bdf7-4c79-ab04-ac7ebc5786e5-kube-api-access-pw7fr\") pod \"authorino-57f7f8659f-dhrl6\" (UID: \"c36e2235-bdf7-4c79-ab04-ac7ebc5786e5\") " pod="kuadrant-system/authorino-57f7f8659f-dhrl6" Nov 03 20:53:47.054499 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:47.054461 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw7fr\" (UniqueName: \"kubernetes.io/projected/c36e2235-bdf7-4c79-ab04-ac7ebc5786e5-kube-api-access-pw7fr\") pod \"authorino-57f7f8659f-dhrl6\" (UID: \"c36e2235-bdf7-4c79-ab04-ac7ebc5786e5\") " pod="kuadrant-system/authorino-57f7f8659f-dhrl6" Nov 03 20:53:47.127334 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:47.127257 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:47.127813 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:47.127625 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw7fr\" (UniqueName: \"kubernetes.io/projected/c36e2235-bdf7-4c79-ab04-ac7ebc5786e5-kube-api-access-pw7fr\") pod \"authorino-57f7f8659f-dhrl6\" (UID: \"c36e2235-bdf7-4c79-ab04-ac7ebc5786e5\") " pod="kuadrant-system/authorino-57f7f8659f-dhrl6" Nov 03 20:53:47.193431 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:47.193396 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-dhrl6" Nov 03 20:53:47.358778 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:47.358745 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:47.559136 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:47.559101 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dhrl6"] Nov 03 20:53:47.565087 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:53:47.565056 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc36e2235_bdf7_4c79_ab04_ac7ebc5786e5.slice/crio-fad3064c31acab7106433d8b01e0e5459979d2b74b3f15f14d94699abc0a75ff WatchSource:0}: Error finding container fad3064c31acab7106433d8b01e0e5459979d2b74b3f15f14d94699abc0a75ff: Status 404 returned error can't find the container with id fad3064c31acab7106433d8b01e0e5459979d2b74b3f15f14d94699abc0a75ff Nov 03 20:53:47.736676 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:47.736635 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-dhrl6" event={"ID":"c36e2235-bdf7-4c79-ab04-ac7ebc5786e5","Type":"ContainerStarted","Data":"fad3064c31acab7106433d8b01e0e5459979d2b74b3f15f14d94699abc0a75ff"} Nov 03 20:53:48.951297 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:48.951267 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:49.135036 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:49.135006 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:49.445115 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:49.445082 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-25r6r"] Nov 03 20:53:49.448552 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:49.448529 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-25r6r" Nov 03 20:53:49.471560 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:49.471529 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47365e7d-f03c-4c02-86f8-8ea1e7b18f10-bound-sa-token\") pod \"cert-manager-597b95cb84-25r6r\" (UID: \"47365e7d-f03c-4c02-86f8-8ea1e7b18f10\") " pod="cert-manager/cert-manager-597b95cb84-25r6r" Nov 03 20:53:49.471735 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:49.471654 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zxb2\" (UniqueName: \"kubernetes.io/projected/47365e7d-f03c-4c02-86f8-8ea1e7b18f10-kube-api-access-4zxb2\") pod \"cert-manager-597b95cb84-25r6r\" (UID: \"47365e7d-f03c-4c02-86f8-8ea1e7b18f10\") " pod="cert-manager/cert-manager-597b95cb84-25r6r" Nov 03 20:53:49.504039 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:49.504003 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-25r6r"] Nov 03 20:53:49.548861 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:49.548828 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-z6lhg\"" Nov 03 20:53:49.572321 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:49.572259 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47365e7d-f03c-4c02-86f8-8ea1e7b18f10-bound-sa-token\") pod \"cert-manager-597b95cb84-25r6r\" (UID: \"47365e7d-f03c-4c02-86f8-8ea1e7b18f10\") " pod="cert-manager/cert-manager-597b95cb84-25r6r" Nov 03 20:53:49.572505 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:49.572397 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zxb2\" (UniqueName: \"kubernetes.io/projected/47365e7d-f03c-4c02-86f8-8ea1e7b18f10-kube-api-access-4zxb2\") pod \"cert-manager-597b95cb84-25r6r\" (UID: \"47365e7d-f03c-4c02-86f8-8ea1e7b18f10\") " pod="cert-manager/cert-manager-597b95cb84-25r6r" Nov 03 20:53:49.630457 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:49.630429 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47365e7d-f03c-4c02-86f8-8ea1e7b18f10-bound-sa-token\") pod \"cert-manager-597b95cb84-25r6r\" (UID: \"47365e7d-f03c-4c02-86f8-8ea1e7b18f10\") " pod="cert-manager/cert-manager-597b95cb84-25r6r" Nov 03 20:53:49.630663 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:49.630638 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zxb2\" (UniqueName: \"kubernetes.io/projected/47365e7d-f03c-4c02-86f8-8ea1e7b18f10-kube-api-access-4zxb2\") pod \"cert-manager-597b95cb84-25r6r\" (UID: \"47365e7d-f03c-4c02-86f8-8ea1e7b18f10\") " pod="cert-manager/cert-manager-597b95cb84-25r6r" Nov 03 20:53:49.761628 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:49.761526 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-25r6r" Nov 03 20:53:50.143648 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:50.143559 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-25r6r"] Nov 03 20:53:50.150512 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:53:50.150479 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47365e7d_f03c_4c02_86f8_8ea1e7b18f10.slice/crio-13ce7c5210f5e37aced96528783d166274ab55beefc26387b0b1d027de053455 WatchSource:0}: Error finding container 13ce7c5210f5e37aced96528783d166274ab55beefc26387b0b1d027de053455: Status 404 returned error can't find the container with id 13ce7c5210f5e37aced96528783d166274ab55beefc26387b0b1d027de053455 Nov 03 20:53:50.752371 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:50.752279 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-dhrl6" event={"ID":"c36e2235-bdf7-4c79-ab04-ac7ebc5786e5","Type":"ContainerStarted","Data":"63c3293255bc4af7a8afca9e689e9fec419206244a2678a2822ebacbdbbf6576"} Nov 03 20:53:50.754048 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:50.754018 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-25r6r" event={"ID":"47365e7d-f03c-4c02-86f8-8ea1e7b18f10","Type":"ContainerStarted","Data":"489802e817defa6e54774465ec440bf32deb8e15b9668670d09be91f67c189f1"} Nov 03 20:53:50.754191 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:50.754056 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-25r6r" event={"ID":"47365e7d-f03c-4c02-86f8-8ea1e7b18f10","Type":"ContainerStarted","Data":"13ce7c5210f5e37aced96528783d166274ab55beefc26387b0b1d027de053455"} Nov 03 20:53:50.861596 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:50.861544 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-dhrl6" podStartSLOduration=1.95285092 podStartE2EDuration="4.861529145s" totalImagesPullingTime="2.908678225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:50.860668893 +0000 UTC m=+380.020741169" watchObservedRunningTime="2026-11-03 20:53:50.861529145 +0000 UTC m=+380.021601424" Nov 03 20:53:50.877931 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:50.877888 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:50.971025 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:50.970979 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-25r6r" podStartSLOduration=1.970963529 podStartE2EDuration="1.970963529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:50.970605969 +0000 UTC m=+380.130678241" watchObservedRunningTime="2026-11-03 20:53:50.970963529 +0000 UTC m=+380.131035801" Nov 03 20:53:52.589580 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:52.589530 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:52.767478 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:52.767428 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:54.065881 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:54.065847 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dhrl6"] Nov 03 20:53:54.066429 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:54.066096 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-dhrl6" podUID="c36e2235-bdf7-4c79-ab04-ac7ebc5786e5" containerName="authorino" containerID="cri-o://63c3293255bc4af7a8afca9e689e9fec419206244a2678a2822ebacbdbbf6576" gracePeriod=30 Nov 03 20:53:54.322383 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:54.322316 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-dhrl6" Nov 03 20:53:54.508124 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:54.508087 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c36e2235-bdf7-4c79-ab04-ac7ebc5786e5-kube-api-access-pw7fr\" (UniqueName: \"kubernetes.io/projected/c36e2235-bdf7-4c79-ab04-ac7ebc5786e5-kube-api-access-pw7fr\") pod \"c36e2235-bdf7-4c79-ab04-ac7ebc5786e5\" (UID: \"c36e2235-bdf7-4c79-ab04-ac7ebc5786e5\") " Nov 03 20:53:54.510218 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:54.510188 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36e2235-bdf7-4c79-ab04-ac7ebc5786e5-kube-api-access-pw7fr" pod "c36e2235-bdf7-4c79-ab04-ac7ebc5786e5" (UID: "c36e2235-bdf7-4c79-ab04-ac7ebc5786e5"). InnerVolumeSpecName "kube-api-access-pw7fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:53:54.609449 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:54.609363 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw7fr\" (UniqueName: \"kubernetes.io/projected/c36e2235-bdf7-4c79-ab04-ac7ebc5786e5-kube-api-access-pw7fr\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:54.772561 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:54.772527 4963 generic.go:356] "Generic (PLEG): container finished" podID="c36e2235-bdf7-4c79-ab04-ac7ebc5786e5" containerID="63c3293255bc4af7a8afca9e689e9fec419206244a2678a2822ebacbdbbf6576" exitCode=0 Nov 03 20:53:54.772775 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:54.772579 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-dhrl6" Nov 03 20:53:54.772775 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:54.772620 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-dhrl6" event={"ID":"c36e2235-bdf7-4c79-ab04-ac7ebc5786e5","Type":"ContainerDied","Data":"63c3293255bc4af7a8afca9e689e9fec419206244a2678a2822ebacbdbbf6576"} Nov 03 20:53:54.772775 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:54.772673 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-dhrl6" event={"ID":"c36e2235-bdf7-4c79-ab04-ac7ebc5786e5","Type":"ContainerDied","Data":"fad3064c31acab7106433d8b01e0e5459979d2b74b3f15f14d94699abc0a75ff"} Nov 03 20:53:54.772775 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:54.772699 4963 scope.go:122] "RemoveContainer" containerID="63c3293255bc4af7a8afca9e689e9fec419206244a2678a2822ebacbdbbf6576" Nov 03 20:53:54.781428 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:54.781409 4963 scope.go:122] "RemoveContainer" containerID="63c3293255bc4af7a8afca9e689e9fec419206244a2678a2822ebacbdbbf6576" Nov 03 20:53:54.781710 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:53:54.781689 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c3293255bc4af7a8afca9e689e9fec419206244a2678a2822ebacbdbbf6576\": container with ID starting with 63c3293255bc4af7a8afca9e689e9fec419206244a2678a2822ebacbdbbf6576 not found: ID does not exist" containerID="63c3293255bc4af7a8afca9e689e9fec419206244a2678a2822ebacbdbbf6576" Nov 03 20:53:54.781788 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:54.781719 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c3293255bc4af7a8afca9e689e9fec419206244a2678a2822ebacbdbbf6576"} err="failed to get container status \"63c3293255bc4af7a8afca9e689e9fec419206244a2678a2822ebacbdbbf6576\": rpc error: code = NotFound desc = could not find container \"63c3293255bc4af7a8afca9e689e9fec419206244a2678a2822ebacbdbbf6576\": container with ID starting with 63c3293255bc4af7a8afca9e689e9fec419206244a2678a2822ebacbdbbf6576 not found: ID does not exist" Nov 03 20:53:54.927817 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:54.927722 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dhrl6"] Nov 03 20:53:54.931851 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:54.931822 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dhrl6"] Nov 03 20:53:54.966982 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:54.966953 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c36e2235-bdf7-4c79-ab04-ac7ebc5786e5" path="/var/lib/kubelet/pods/c36e2235-bdf7-4c79-ab04-ac7ebc5786e5/volumes" Nov 03 20:53:55.592596 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:55.592559 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:55.769998 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:53:55.769962 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:01.198854 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:01.198820 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:01.422995 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:01.422940 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:08.356456 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:08.356422 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:08.544349 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:08.544314 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:09.145760 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:09.145728 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:12.527622 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:54:12.527541 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:54:22.418693 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:22.418656 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:22.599442 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:22.599416 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:22.779530 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:22.779474 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:23.296875 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:23.296832 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:25.478558 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:25.478487 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:25.656546 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:25.656502 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:32.472490 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.472424 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:32.509450 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.509411 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58f54db7dc-9xjv6"] Nov 03 20:54:32.510001 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.509976 4963 cpu_manager.go:414] "containerMap: removing container" podUID="c36e2235-bdf7-4c79-ab04-ac7ebc5786e5" containerName="authorino" Nov 03 20:54:32.510001 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.509999 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c36e2235-bdf7-4c79-ab04-ac7ebc5786e5" containerName="authorino" Nov 03 20:54:32.510189 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.510149 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="c36e2235-bdf7-4c79-ab04-ac7ebc5786e5" containerName="authorino" Nov 03 20:54:32.517771 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.517748 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f54db7dc-9xjv6" Nov 03 20:54:32.568386 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.568356 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 20:54:32.568984 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.568962 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58f54db7dc-9xjv6"] Nov 03 20:54:32.614227 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.614190 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 20:54:32.614227 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.614190 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 20:54:32.614432 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.614237 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 20:54:32.614432 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.614201 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 20:54:32.614597 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.614581 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9vrxc\"" Nov 03 20:54:32.639975 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.639900 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p269f\" (UniqueName: \"kubernetes.io/projected/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-kube-api-access-p269f\") pod \"maas-controller-58f54db7dc-9xjv6\" (UID: \"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9\") " pod="opendatahub/maas-controller-58f54db7dc-9xjv6" Nov 03 20:54:32.640159 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.640044 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-metrics-certs\") pod \"maas-controller-58f54db7dc-9xjv6\" (UID: \"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9\") " pod="opendatahub/maas-controller-58f54db7dc-9xjv6" Nov 03 20:54:32.640159 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.640092 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-webhook-cert\") pod \"maas-controller-58f54db7dc-9xjv6\" (UID: \"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9\") " pod="opendatahub/maas-controller-58f54db7dc-9xjv6" Nov 03 20:54:32.653263 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.653233 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:32.740831 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.740743 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p269f\" (UniqueName: \"kubernetes.io/projected/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-kube-api-access-p269f\") pod \"maas-controller-58f54db7dc-9xjv6\" (UID: \"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9\") " pod="opendatahub/maas-controller-58f54db7dc-9xjv6" Nov 03 20:54:32.740831 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.740796 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-metrics-certs\") pod \"maas-controller-58f54db7dc-9xjv6\" (UID: \"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9\") " pod="opendatahub/maas-controller-58f54db7dc-9xjv6" Nov 03 20:54:32.740831 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.740823 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-webhook-cert\") pod \"maas-controller-58f54db7dc-9xjv6\" (UID: \"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9\") " pod="opendatahub/maas-controller-58f54db7dc-9xjv6" Nov 03 20:54:32.743392 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.743363 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-metrics-certs\") pod \"maas-controller-58f54db7dc-9xjv6\" (UID: \"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9\") " pod="opendatahub/maas-controller-58f54db7dc-9xjv6" Nov 03 20:54:32.743505 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.743432 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-webhook-cert\") pod \"maas-controller-58f54db7dc-9xjv6\" (UID: \"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9\") " pod="opendatahub/maas-controller-58f54db7dc-9xjv6" Nov 03 20:54:32.799241 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.799203 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p269f\" (UniqueName: \"kubernetes.io/projected/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-kube-api-access-p269f\") pod \"maas-controller-58f54db7dc-9xjv6\" (UID: \"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9\") " pod="opendatahub/maas-controller-58f54db7dc-9xjv6" Nov 03 20:54:32.833093 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.833055 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f54db7dc-9xjv6" Nov 03 20:54:32.837244 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:32.837219 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:33.017235 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:33.017148 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:33.126487 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:33.126446 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77b845648-gxp5x"] Nov 03 20:54:33.134965 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:33.134934 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77b845648-gxp5x" Nov 03 20:54:33.185583 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:33.185545 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77b845648-gxp5x"] Nov 03 20:54:33.246196 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:33.246162 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5fac629-6443-4e4e-a5b3-08a949199c32-metrics-certs\") pod \"maas-controller-77b845648-gxp5x\" (UID: \"e5fac629-6443-4e4e-a5b3-08a949199c32\") " pod="opendatahub/maas-controller-77b845648-gxp5x" Nov 03 20:54:33.246378 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:33.246212 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e5fac629-6443-4e4e-a5b3-08a949199c32-webhook-cert\") pod \"maas-controller-77b845648-gxp5x\" (UID: \"e5fac629-6443-4e4e-a5b3-08a949199c32\") " pod="opendatahub/maas-controller-77b845648-gxp5x" Nov 03 20:54:33.246428 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:33.246371 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2p72\" (UniqueName: \"kubernetes.io/projected/e5fac629-6443-4e4e-a5b3-08a949199c32-kube-api-access-p2p72\") pod \"maas-controller-77b845648-gxp5x\" (UID: \"e5fac629-6443-4e4e-a5b3-08a949199c32\") " pod="opendatahub/maas-controller-77b845648-gxp5x" Nov 03 20:54:33.249709 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:33.249686 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58f54db7dc-9xjv6"] Nov 03 20:54:33.255749 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:54:33.255724 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d32cb35_3df8_4763_9e74_0dbaf7fcf7d9.slice/crio-287b9f0de183c4382ac49e6dc91b4d3238b912fd35355f79cd0dc378467dfe6c WatchSource:0}: Error finding container 287b9f0de183c4382ac49e6dc91b4d3238b912fd35355f79cd0dc378467dfe6c: Status 404 returned error can't find the container with id 287b9f0de183c4382ac49e6dc91b4d3238b912fd35355f79cd0dc378467dfe6c Nov 03 20:54:33.347087 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:33.346994 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2p72\" (UniqueName: \"kubernetes.io/projected/e5fac629-6443-4e4e-a5b3-08a949199c32-kube-api-access-p2p72\") pod \"maas-controller-77b845648-gxp5x\" (UID: \"e5fac629-6443-4e4e-a5b3-08a949199c32\") " pod="opendatahub/maas-controller-77b845648-gxp5x" Nov 03 20:54:33.347087 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:33.347046 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5fac629-6443-4e4e-a5b3-08a949199c32-metrics-certs\") pod \"maas-controller-77b845648-gxp5x\" (UID: \"e5fac629-6443-4e4e-a5b3-08a949199c32\") " pod="opendatahub/maas-controller-77b845648-gxp5x" Nov 03 20:54:33.347337 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:33.347095 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e5fac629-6443-4e4e-a5b3-08a949199c32-webhook-cert\") pod \"maas-controller-77b845648-gxp5x\" (UID: \"e5fac629-6443-4e4e-a5b3-08a949199c32\") " pod="opendatahub/maas-controller-77b845648-gxp5x" Nov 03 20:54:33.349721 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:33.349691 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e5fac629-6443-4e4e-a5b3-08a949199c32-webhook-cert\") pod \"maas-controller-77b845648-gxp5x\" (UID: \"e5fac629-6443-4e4e-a5b3-08a949199c32\") " pod="opendatahub/maas-controller-77b845648-gxp5x" Nov 03 20:54:33.349833 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:33.349744 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5fac629-6443-4e4e-a5b3-08a949199c32-metrics-certs\") pod \"maas-controller-77b845648-gxp5x\" (UID: \"e5fac629-6443-4e4e-a5b3-08a949199c32\") " pod="opendatahub/maas-controller-77b845648-gxp5x" Nov 03 20:54:33.405728 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:33.405693 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2p72\" (UniqueName: \"kubernetes.io/projected/e5fac629-6443-4e4e-a5b3-08a949199c32-kube-api-access-p2p72\") pod \"maas-controller-77b845648-gxp5x\" (UID: \"e5fac629-6443-4e4e-a5b3-08a949199c32\") " pod="opendatahub/maas-controller-77b845648-gxp5x" Nov 03 20:54:33.447897 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:33.447851 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77b845648-gxp5x" Nov 03 20:54:33.819047 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:33.819006 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77b845648-gxp5x"] Nov 03 20:54:33.824809 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:54:33.824778 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5fac629_6443_4e4e_a5b3_08a949199c32.slice/crio-a3940ce6df687c77217555c4a73dc608a32a83a92511626c71b2acd96b125f4f WatchSource:0}: Error finding container a3940ce6df687c77217555c4a73dc608a32a83a92511626c71b2acd96b125f4f: Status 404 returned error can't find the container with id a3940ce6df687c77217555c4a73dc608a32a83a92511626c71b2acd96b125f4f Nov 03 20:54:33.951950 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:33.951882 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f54db7dc-9xjv6" event={"ID":"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9","Type":"ContainerStarted","Data":"287b9f0de183c4382ac49e6dc91b4d3238b912fd35355f79cd0dc378467dfe6c"} Nov 03 20:54:33.953115 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:33.953088 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77b845648-gxp5x" event={"ID":"e5fac629-6443-4e4e-a5b3-08a949199c32","Type":"ContainerStarted","Data":"a3940ce6df687c77217555c4a73dc608a32a83a92511626c71b2acd96b125f4f"} Nov 03 20:54:34.890211 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:34.890174 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:37.342046 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:37.342014 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:37.525380 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:37.525333 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:39.931314 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:39.931212 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:39.991281 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:39.991215 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77b845648-gxp5x" event={"ID":"e5fac629-6443-4e4e-a5b3-08a949199c32","Type":"ContainerStarted","Data":"2a8fbd5a72201e13aa47440a034cc7166037b735ef374235b1427077992f3629"} Nov 03 20:54:39.991467 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:39.991369 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77b845648-gxp5x" Nov 03 20:54:39.992967 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:39.992932 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f54db7dc-9xjv6" event={"ID":"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9","Type":"ContainerStarted","Data":"5b60a7df355fbe3c1357a9c9e25f295ea1c6b3eee040b449aa481e31fbd50a2b"} Nov 03 20:54:39.993076 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:39.993062 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58f54db7dc-9xjv6" Nov 03 20:54:40.104572 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:40.104501 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-77b845648-gxp5x" podStartSLOduration=1.433097561 podStartE2EDuration="7.104487042s" totalImagesPullingTime="5.671389481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:40.103583613 +0000 UTC m=+429.263655897" watchObservedRunningTime="2026-11-03 20:54:40.104487042 +0000 UTC m=+429.264559313" Nov 03 20:54:40.123105 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:40.123071 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:40.217663 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:40.217602 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-58f54db7dc-9xjv6" podStartSLOduration=1.967692656 podStartE2EDuration="8.217586822s" totalImagesPullingTime="6.249894166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:40.215927678 +0000 UTC m=+429.375999941" watchObservedRunningTime="2026-11-03 20:54:40.217586822 +0000 UTC m=+429.377659143" Nov 03 20:54:42.528070 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:54:42.528023 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:54:42.704462 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.704427 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5"] Nov 03 20:54:42.707768 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.707745 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" Nov 03 20:54:42.759518 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.759487 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-qtgzp\"" Nov 03 20:54:42.760143 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.760120 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 20:54:42.806897 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.806818 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 20:54:42.808998 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.808976 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5"] Nov 03 20:54:42.836167 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.836130 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pklw\" (UniqueName: \"kubernetes.io/projected/760f65bf-694d-407a-888b-c408028109f6-kube-api-access-6pklw\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gdzc5\" (UID: \"760f65bf-694d-407a-888b-c408028109f6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" Nov 03 20:54:42.836380 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.836255 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/760f65bf-694d-407a-888b-c408028109f6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gdzc5\" (UID: \"760f65bf-694d-407a-888b-c408028109f6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" Nov 03 20:54:42.836380 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.836315 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/760f65bf-694d-407a-888b-c408028109f6-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gdzc5\" (UID: \"760f65bf-694d-407a-888b-c408028109f6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" Nov 03 20:54:42.836380 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.836359 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/760f65bf-694d-407a-888b-c408028109f6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gdzc5\" (UID: \"760f65bf-694d-407a-888b-c408028109f6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" Nov 03 20:54:42.861339 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.861306 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 20:54:42.883094 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.883062 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:42.937393 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.937352 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pklw\" (UniqueName: \"kubernetes.io/projected/760f65bf-694d-407a-888b-c408028109f6-kube-api-access-6pklw\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gdzc5\" (UID: \"760f65bf-694d-407a-888b-c408028109f6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" Nov 03 20:54:42.937591 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.937429 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/760f65bf-694d-407a-888b-c408028109f6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gdzc5\" (UID: \"760f65bf-694d-407a-888b-c408028109f6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" Nov 03 20:54:42.937591 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.937465 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/760f65bf-694d-407a-888b-c408028109f6-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gdzc5\" (UID: \"760f65bf-694d-407a-888b-c408028109f6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" Nov 03 20:54:42.937591 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.937510 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/760f65bf-694d-407a-888b-c408028109f6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gdzc5\" (UID: \"760f65bf-694d-407a-888b-c408028109f6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" Nov 03 20:54:42.937591 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:54:42.937545 4963 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 03 20:54:42.937803 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:54:42.937617 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/760f65bf-694d-407a-888b-c408028109f6-proxy-source podName:760f65bf-694d-407a-888b-c408028109f6 nodeName:}" failed. No retries permitted until 2026-11-03 20:54:43.437594995 +0000 UTC m=+432.597667292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/760f65bf-694d-407a-888b-c408028109f6-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" (UID: "760f65bf-694d-407a-888b-c408028109f6") : configmap "usage-tenancy-proxy-source" not found Nov 03 20:54:42.938242 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.938216 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/760f65bf-694d-407a-888b-c408028109f6-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gdzc5\" (UID: \"760f65bf-694d-407a-888b-c408028109f6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" Nov 03 20:54:42.940141 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.940119 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/760f65bf-694d-407a-888b-c408028109f6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gdzc5\" (UID: \"760f65bf-694d-407a-888b-c408028109f6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" Nov 03 20:54:42.998198 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:42.998165 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pklw\" (UniqueName: \"kubernetes.io/projected/760f65bf-694d-407a-888b-c408028109f6-kube-api-access-6pklw\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gdzc5\" (UID: \"760f65bf-694d-407a-888b-c408028109f6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" Nov 03 20:54:43.067139 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:43.067047 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:43.443353 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:43.443316 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/760f65bf-694d-407a-888b-c408028109f6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gdzc5\" (UID: \"760f65bf-694d-407a-888b-c408028109f6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" Nov 03 20:54:43.444142 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:43.444123 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/760f65bf-694d-407a-888b-c408028109f6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gdzc5\" (UID: \"760f65bf-694d-407a-888b-c408028109f6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" Nov 03 20:54:43.620762 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:43.620721 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" Nov 03 20:54:43.989555 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:43.989521 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5"] Nov 03 20:54:43.995852 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:54:43.995819 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760f65bf_694d_407a_888b_c408028109f6.slice/crio-ffe24da69e75e7da885c0d8669bd96132d6ddc42ad6e143a01c17e7f0d2fe3e7 WatchSource:0}: Error finding container ffe24da69e75e7da885c0d8669bd96132d6ddc42ad6e143a01c17e7f0d2fe3e7: Status 404 returned error can't find the container with id ffe24da69e75e7da885c0d8669bd96132d6ddc42ad6e143a01c17e7f0d2fe3e7 Nov 03 20:54:44.013879 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:44.013845 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" event={"ID":"760f65bf-694d-407a-888b-c408028109f6","Type":"ContainerStarted","Data":"ffe24da69e75e7da885c0d8669bd96132d6ddc42ad6e143a01c17e7f0d2fe3e7"} Nov 03 20:54:51.001659 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:51.001626 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77b845648-gxp5x" Nov 03 20:54:51.002180 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:51.001711 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58f54db7dc-9xjv6" Nov 03 20:54:51.304508 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:51.304404 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58f54db7dc-9xjv6"] Nov 03 20:54:51.304843 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:51.304634 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-58f54db7dc-9xjv6" podUID="0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9" containerName="manager" containerID="cri-o://5b60a7df355fbe3c1357a9c9e25f295ea1c6b3eee040b449aa481e31fbd50a2b" gracePeriod=30 Nov 03 20:54:51.599531 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:51.599497 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f54db7dc-9xjv6" Nov 03 20:54:51.640863 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:51.640825 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-metrics-certs\") pod \"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9\" (UID: \"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9\") " Nov 03 20:54:51.641078 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:51.640890 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-kube-api-access-p269f\" (UniqueName: \"kubernetes.io/projected/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-kube-api-access-p269f\") pod \"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9\" (UID: \"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9\") " Nov 03 20:54:51.641078 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:51.640945 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-webhook-cert\") pod \"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9\" (UID: \"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9\") " Nov 03 20:54:51.644398 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:51.644361 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-metrics-certs" pod "0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9" (UID: "0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:51.644555 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:51.644452 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-webhook-cert" pod "0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9" (UID: "0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:51.644555 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:51.644501 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-kube-api-access-p269f" pod "0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9" (UID: "0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9"). InnerVolumeSpecName "kube-api-access-p269f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:51.742345 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:51.742266 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:51.742345 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:51.742305 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p269f\" (UniqueName: \"kubernetes.io/projected/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-kube-api-access-p269f\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:51.742345 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:51.742320 4963 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9-webhook-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:52.061689 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:52.061658 4963 generic.go:356] "Generic (PLEG): container finished" podID="0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9" containerID="5b60a7df355fbe3c1357a9c9e25f295ea1c6b3eee040b449aa481e31fbd50a2b" exitCode=0 Nov 03 20:54:52.062151 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:52.061713 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f54db7dc-9xjv6" Nov 03 20:54:52.062151 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:52.061749 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f54db7dc-9xjv6" event={"ID":"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9","Type":"ContainerDied","Data":"5b60a7df355fbe3c1357a9c9e25f295ea1c6b3eee040b449aa481e31fbd50a2b"} Nov 03 20:54:52.062151 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:52.061796 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f54db7dc-9xjv6" event={"ID":"0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9","Type":"ContainerDied","Data":"287b9f0de183c4382ac49e6dc91b4d3238b912fd35355f79cd0dc378467dfe6c"} Nov 03 20:54:52.062151 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:52.061816 4963 scope.go:122] "RemoveContainer" containerID="5b60a7df355fbe3c1357a9c9e25f295ea1c6b3eee040b449aa481e31fbd50a2b" Nov 03 20:54:52.072019 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:52.071995 4963 scope.go:122] "RemoveContainer" containerID="5b60a7df355fbe3c1357a9c9e25f295ea1c6b3eee040b449aa481e31fbd50a2b" Nov 03 20:54:52.072303 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:54:52.072281 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b60a7df355fbe3c1357a9c9e25f295ea1c6b3eee040b449aa481e31fbd50a2b\": container with ID starting with 5b60a7df355fbe3c1357a9c9e25f295ea1c6b3eee040b449aa481e31fbd50a2b not found: ID does not exist" containerID="5b60a7df355fbe3c1357a9c9e25f295ea1c6b3eee040b449aa481e31fbd50a2b" Nov 03 20:54:52.072369 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:52.072316 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b60a7df355fbe3c1357a9c9e25f295ea1c6b3eee040b449aa481e31fbd50a2b"} err="failed to get container status \"5b60a7df355fbe3c1357a9c9e25f295ea1c6b3eee040b449aa481e31fbd50a2b\": rpc error: code = NotFound desc = could not find container \"5b60a7df355fbe3c1357a9c9e25f295ea1c6b3eee040b449aa481e31fbd50a2b\": container with ID starting with 5b60a7df355fbe3c1357a9c9e25f295ea1c6b3eee040b449aa481e31fbd50a2b not found: ID does not exist" Nov 03 20:54:52.220538 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:52.220503 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58f54db7dc-9xjv6"] Nov 03 20:54:52.267578 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:52.267544 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58f54db7dc-9xjv6"] Nov 03 20:54:52.967155 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:52.967117 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9" path="/var/lib/kubelet/pods/0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9/volumes" Nov 03 20:54:54.812353 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:54.812284 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:54.994864 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:54.994796 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:57.092094 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:57.092059 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" event={"ID":"760f65bf-694d-407a-888b-c408028109f6","Type":"ContainerStarted","Data":"022fb7f0f6d0b4035707b400931b9040e949367424ebc5cef323fed250b7c33c"} Nov 03 20:54:57.092595 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:57.092170 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" Nov 03 20:54:57.203581 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:54:57.203531 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" podStartSLOduration=2.209501682 podStartE2EDuration="15.203518376s" totalImagesPullingTime="12.994016694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:57.203095368 +0000 UTC m=+446.363167639" watchObservedRunningTime="2026-11-03 20:54:57.203518376 +0000 UTC m=+446.363590647" Nov 03 20:55:02.311371 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:02.311335 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:02.494241 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:02.494210 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:02.677818 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:02.677718 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:02.856995 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:02.856953 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:03.104756 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.104723 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:03.290032 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.289987 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:03.341856 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.341815 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n"] Nov 03 20:55:03.342426 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.342405 4963 cpu_manager.go:414] "containerMap: removing container" podUID="0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9" containerName="manager" Nov 03 20:55:03.342502 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.342429 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9" containerName="manager" Nov 03 20:55:03.342590 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.342577 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d32cb35-3df8-4763-9e74-0dbaf7fcf7d9" containerName="manager" Nov 03 20:55:03.455360 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.455316 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n"] Nov 03 20:55:03.455545 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.455429 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n" Nov 03 20:55:03.505474 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.505444 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-7chnt\"" Nov 03 20:55:03.505474 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.505451 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 20:55:03.531772 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.531729 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:03.552980 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.552939 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7lll\" (UniqueName: \"kubernetes.io/projected/26c4944c-af22-41af-8270-ad1e85513dab-kube-api-access-n7lll\") pod \"llmisvc-controller-manager-869d5dcc5d-5wz4n\" (UID: \"26c4944c-af22-41af-8270-ad1e85513dab\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n" Nov 03 20:55:03.552980 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.552984 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26c4944c-af22-41af-8270-ad1e85513dab-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5wz4n\" (UID: \"26c4944c-af22-41af-8270-ad1e85513dab\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n" Nov 03 20:55:03.577515 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.577476 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-jww6w"] Nov 03 20:55:03.606679 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.606646 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-jww6w" Nov 03 20:55:03.636509 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.636468 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-jww6w"] Nov 03 20:55:03.654489 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.654447 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7lll\" (UniqueName: \"kubernetes.io/projected/26c4944c-af22-41af-8270-ad1e85513dab-kube-api-access-n7lll\") pod \"llmisvc-controller-manager-869d5dcc5d-5wz4n\" (UID: \"26c4944c-af22-41af-8270-ad1e85513dab\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n" Nov 03 20:55:03.654489 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.654489 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26c4944c-af22-41af-8270-ad1e85513dab-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5wz4n\" (UID: \"26c4944c-af22-41af-8270-ad1e85513dab\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n" Nov 03 20:55:03.654858 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.654519 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mnn4\" (UniqueName: \"kubernetes.io/projected/fd4d8972-0ca6-40d6-8bd7-993d611500b4-kube-api-access-4mnn4\") pod \"odh-model-controller-769875db56-jww6w\" (UID: \"fd4d8972-0ca6-40d6-8bd7-993d611500b4\") " pod="opendatahub/odh-model-controller-769875db56-jww6w" Nov 03 20:55:03.654858 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.654569 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd4d8972-0ca6-40d6-8bd7-993d611500b4-cert\") pod \"odh-model-controller-769875db56-jww6w\" (UID: \"fd4d8972-0ca6-40d6-8bd7-993d611500b4\") " pod="opendatahub/odh-model-controller-769875db56-jww6w" Nov 03 20:55:03.654858 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.654596 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd4d8972-0ca6-40d6-8bd7-993d611500b4-metrics-certs\") pod \"odh-model-controller-769875db56-jww6w\" (UID: \"fd4d8972-0ca6-40d6-8bd7-993d611500b4\") " pod="opendatahub/odh-model-controller-769875db56-jww6w" Nov 03 20:55:03.657412 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.657381 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26c4944c-af22-41af-8270-ad1e85513dab-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5wz4n\" (UID: \"26c4944c-af22-41af-8270-ad1e85513dab\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n" Nov 03 20:55:03.660372 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.660345 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-bvw8t\"" Nov 03 20:55:03.660494 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.660394 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 20:55:03.660494 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.660442 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 20:55:03.713192 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.713115 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7lll\" (UniqueName: \"kubernetes.io/projected/26c4944c-af22-41af-8270-ad1e85513dab-kube-api-access-n7lll\") pod \"llmisvc-controller-manager-869d5dcc5d-5wz4n\" (UID: \"26c4944c-af22-41af-8270-ad1e85513dab\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n" Nov 03 20:55:03.755224 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.755185 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd4d8972-0ca6-40d6-8bd7-993d611500b4-cert\") pod \"odh-model-controller-769875db56-jww6w\" (UID: \"fd4d8972-0ca6-40d6-8bd7-993d611500b4\") " pod="opendatahub/odh-model-controller-769875db56-jww6w" Nov 03 20:55:03.755224 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.755235 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd4d8972-0ca6-40d6-8bd7-993d611500b4-metrics-certs\") pod \"odh-model-controller-769875db56-jww6w\" (UID: \"fd4d8972-0ca6-40d6-8bd7-993d611500b4\") " pod="opendatahub/odh-model-controller-769875db56-jww6w" Nov 03 20:55:03.755466 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.755329 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mnn4\" (UniqueName: \"kubernetes.io/projected/fd4d8972-0ca6-40d6-8bd7-993d611500b4-kube-api-access-4mnn4\") pod \"odh-model-controller-769875db56-jww6w\" (UID: \"fd4d8972-0ca6-40d6-8bd7-993d611500b4\") " pod="opendatahub/odh-model-controller-769875db56-jww6w" Nov 03 20:55:03.757699 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.757676 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd4d8972-0ca6-40d6-8bd7-993d611500b4-cert\") pod \"odh-model-controller-769875db56-jww6w\" (UID: \"fd4d8972-0ca6-40d6-8bd7-993d611500b4\") " pod="opendatahub/odh-model-controller-769875db56-jww6w" Nov 03 20:55:03.757810 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.757788 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd4d8972-0ca6-40d6-8bd7-993d611500b4-metrics-certs\") pod \"odh-model-controller-769875db56-jww6w\" (UID: \"fd4d8972-0ca6-40d6-8bd7-993d611500b4\") " pod="opendatahub/odh-model-controller-769875db56-jww6w" Nov 03 20:55:03.790450 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.790413 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n" Nov 03 20:55:03.813542 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.813503 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mnn4\" (UniqueName: \"kubernetes.io/projected/fd4d8972-0ca6-40d6-8bd7-993d611500b4-kube-api-access-4mnn4\") pod \"odh-model-controller-769875db56-jww6w\" (UID: \"fd4d8972-0ca6-40d6-8bd7-993d611500b4\") " pod="opendatahub/odh-model-controller-769875db56-jww6w" Nov 03 20:55:03.936546 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:03.936500 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-jww6w" Nov 03 20:55:04.191294 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:04.191248 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n"] Nov 03 20:55:04.197518 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:55:04.197477 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod26c4944c_af22_41af_8270_ad1e85513dab.slice/crio-52f9296b1356c4475becb7d9d0a1b72933f0582df31b69f64dd7e2495d85fb71 WatchSource:0}: Error finding container 52f9296b1356c4475becb7d9d0a1b72933f0582df31b69f64dd7e2495d85fb71: Status 404 returned error can't find the container with id 52f9296b1356c4475becb7d9d0a1b72933f0582df31b69f64dd7e2495d85fb71 Nov 03 20:55:04.335094 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:04.335021 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-jww6w"] Nov 03 20:55:04.341129 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:55:04.341098 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4d8972_0ca6_40d6_8bd7_993d611500b4.slice/crio-8055192bee62af211eb1161461ab24df14ce0cf7472fa81ec815a079da8bb092 WatchSource:0}: Error finding container 8055192bee62af211eb1161461ab24df14ce0cf7472fa81ec815a079da8bb092: Status 404 returned error can't find the container with id 8055192bee62af211eb1161461ab24df14ce0cf7472fa81ec815a079da8bb092 Nov 03 20:55:04.680951 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:04.680857 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:05.131705 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:05.131671 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n" event={"ID":"26c4944c-af22-41af-8270-ad1e85513dab","Type":"ContainerStarted","Data":"52f9296b1356c4475becb7d9d0a1b72933f0582df31b69f64dd7e2495d85fb71"} Nov 03 20:55:05.132816 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:05.132793 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-jww6w" event={"ID":"fd4d8972-0ca6-40d6-8bd7-993d611500b4","Type":"ContainerStarted","Data":"8055192bee62af211eb1161461ab24df14ce0cf7472fa81ec815a079da8bb092"} Nov 03 20:55:08.103678 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:08.103649 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gdzc5" Nov 03 20:55:08.153062 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:08.153027 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n" event={"ID":"26c4944c-af22-41af-8270-ad1e85513dab","Type":"ContainerStarted","Data":"37ba629e890a073ed2586dd42e313e2610f3570d389984a51e4b70cfda1c1de1"} Nov 03 20:55:08.153221 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:08.153083 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n" Nov 03 20:55:08.326089 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:08.326032 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n" podStartSLOduration=1.9566434209999999 podStartE2EDuration="5.326016107s" totalImagesPullingTime="3.369372686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:08.324405133 +0000 UTC m=+457.484477404" watchObservedRunningTime="2026-11-03 20:55:08.326016107 +0000 UTC m=+457.486088379" Nov 03 20:55:09.159962 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:09.159918 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-jww6w" event={"ID":"fd4d8972-0ca6-40d6-8bd7-993d611500b4","Type":"ContainerStarted","Data":"bd699645e0e8ab96f8896769c603f7df66bf2bcc9cc6a2a072bd41985beaafed"} Nov 03 20:55:09.160394 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:09.160079 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-jww6w" Nov 03 20:55:09.273225 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:09.273172 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-jww6w" podStartSLOduration=2.494803057 podStartE2EDuration="6.273157602s" totalImagesPullingTime="3.778354545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:09.271313844 +0000 UTC m=+458.431386120" watchObservedRunningTime="2026-11-03 20:55:09.273157602 +0000 UTC m=+458.433229874" Nov 03 20:55:09.972300 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:09.972265 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:10.157243 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:10.157214 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:10.339710 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:10.339626 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:10.517470 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:10.517435 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:10.702532 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:10.702495 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:10.886118 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:10.886078 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:11.064136 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:11.064054 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:11.253892 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:11.253862 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:12.526543 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:55:12.526509 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:55:13.107512 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:13.107470 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:13.292317 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:13.292282 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:15.211574 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:15.211541 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:15.396726 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:15.396695 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:15.580655 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:15.580571 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:15.761049 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:15.761015 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:16.281543 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.281509 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw"] Nov 03 20:55:16.284261 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.284241 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" Nov 03 20:55:16.377463 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.377429 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699b9513-7af2-4d80-9f67-105257832239-openshift-service-ca\") pod \"maas-api-6dfd88bddd-xgckw\" (UID: \"699b9513-7af2-4d80-9f67-105257832239\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" Nov 03 20:55:16.377463 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.377464 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm9qg\" (UniqueName: \"kubernetes.io/projected/699b9513-7af2-4d80-9f67-105257832239-kube-api-access-nm9qg\") pod \"maas-api-6dfd88bddd-xgckw\" (UID: \"699b9513-7af2-4d80-9f67-105257832239\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" Nov 03 20:55:16.377685 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.377598 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699b9513-7af2-4d80-9f67-105257832239-maas-api-tls\") pod \"maas-api-6dfd88bddd-xgckw\" (UID: \"699b9513-7af2-4d80-9f67-105257832239\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" Nov 03 20:55:16.377685 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.377631 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/699b9513-7af2-4d80-9f67-105257832239-metrics-certs\") pod \"maas-api-6dfd88bddd-xgckw\" (UID: \"699b9513-7af2-4d80-9f67-105257832239\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" Nov 03 20:55:16.381539 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.381509 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 20:55:16.381539 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.381513 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:55:16.381719 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.381539 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6whcf\"" Nov 03 20:55:16.381719 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.381579 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:55:16.381719 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.381628 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:55:16.383693 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.383675 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw"] Nov 03 20:55:16.478279 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.478237 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699b9513-7af2-4d80-9f67-105257832239-openshift-service-ca\") pod \"maas-api-6dfd88bddd-xgckw\" (UID: \"699b9513-7af2-4d80-9f67-105257832239\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" Nov 03 20:55:16.478279 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.478274 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm9qg\" (UniqueName: \"kubernetes.io/projected/699b9513-7af2-4d80-9f67-105257832239-kube-api-access-nm9qg\") pod \"maas-api-6dfd88bddd-xgckw\" (UID: \"699b9513-7af2-4d80-9f67-105257832239\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" Nov 03 20:55:16.478620 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.478335 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699b9513-7af2-4d80-9f67-105257832239-maas-api-tls\") pod \"maas-api-6dfd88bddd-xgckw\" (UID: \"699b9513-7af2-4d80-9f67-105257832239\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" Nov 03 20:55:16.478620 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.478359 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/699b9513-7af2-4d80-9f67-105257832239-metrics-certs\") pod \"maas-api-6dfd88bddd-xgckw\" (UID: \"699b9513-7af2-4d80-9f67-105257832239\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" Nov 03 20:55:16.479030 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.479003 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699b9513-7af2-4d80-9f67-105257832239-openshift-service-ca\") pod \"maas-api-6dfd88bddd-xgckw\" (UID: \"699b9513-7af2-4d80-9f67-105257832239\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" Nov 03 20:55:16.481042 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.481018 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/699b9513-7af2-4d80-9f67-105257832239-metrics-certs\") pod \"maas-api-6dfd88bddd-xgckw\" (UID: \"699b9513-7af2-4d80-9f67-105257832239\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" Nov 03 20:55:16.481126 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.481021 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699b9513-7af2-4d80-9f67-105257832239-maas-api-tls\") pod \"maas-api-6dfd88bddd-xgckw\" (UID: \"699b9513-7af2-4d80-9f67-105257832239\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" Nov 03 20:55:16.539552 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.539485 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm9qg\" (UniqueName: \"kubernetes.io/projected/699b9513-7af2-4d80-9f67-105257832239-kube-api-access-nm9qg\") pod \"maas-api-6dfd88bddd-xgckw\" (UID: \"699b9513-7af2-4d80-9f67-105257832239\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" Nov 03 20:55:16.596082 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.596043 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" Nov 03 20:55:16.968120 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:16.968091 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw"] Nov 03 20:55:16.974415 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:55:16.974387 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699b9513_7af2_4d80_9f67_105257832239.slice/crio-d49b1fd70040458d3671a8a1d96b846516aa428c037b10188c39358c33b40851 WatchSource:0}: Error finding container d49b1fd70040458d3671a8a1d96b846516aa428c037b10188c39358c33b40851: Status 404 returned error can't find the container with id d49b1fd70040458d3671a8a1d96b846516aa428c037b10188c39358c33b40851 Nov 03 20:55:17.199207 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:17.199173 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" event={"ID":"699b9513-7af2-4d80-9f67-105257832239","Type":"ContainerStarted","Data":"d49b1fd70040458d3671a8a1d96b846516aa428c037b10188c39358c33b40851"} Nov 03 20:55:20.166886 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:20.166856 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-jww6w" Nov 03 20:55:23.632745 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:23.632703 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:24.235934 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:24.235871 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" event={"ID":"699b9513-7af2-4d80-9f67-105257832239","Type":"ContainerStarted","Data":"b424a7a49b14323266671ea23b62d77beb9b530a46637424ad27e0d109d86612"} Nov 03 20:55:24.236111 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:24.235952 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" Nov 03 20:55:24.348367 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:24.348310 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" podStartSLOduration=1.548113153 podStartE2EDuration="8.348297223s" totalImagesPullingTime="6.80018407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:24.347729726 +0000 UTC m=+473.507801997" watchObservedRunningTime="2026-11-03 20:55:24.348297223 +0000 UTC m=+473.508369495" Nov 03 20:55:30.248291 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:30.248258 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" Nov 03 20:55:30.801877 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:30.801829 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:30.979533 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:30.979504 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:31.078189 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:31.078110 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lt4xd"] Nov 03 20:55:31.081403 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:31.081377 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lt4xd" Nov 03 20:55:31.132881 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:31.132849 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dq8fb\"" Nov 03 20:55:31.134949 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:31.134927 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lt4xd"] Nov 03 20:55:31.212629 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:31.212586 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lr8d\" (UniqueName: \"kubernetes.io/projected/925666af-27fd-41b8-aff2-56de3a2071a8-kube-api-access-4lr8d\") pod \"authorino-6f6944bcdd-lt4xd\" (UID: \"925666af-27fd-41b8-aff2-56de3a2071a8\") " pod="kuadrant-system/authorino-6f6944bcdd-lt4xd" Nov 03 20:55:31.313782 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:31.313744 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lr8d\" (UniqueName: \"kubernetes.io/projected/925666af-27fd-41b8-aff2-56de3a2071a8-kube-api-access-4lr8d\") pod \"authorino-6f6944bcdd-lt4xd\" (UID: \"925666af-27fd-41b8-aff2-56de3a2071a8\") " pod="kuadrant-system/authorino-6f6944bcdd-lt4xd" Nov 03 20:55:31.373483 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:31.373413 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lr8d\" (UniqueName: \"kubernetes.io/projected/925666af-27fd-41b8-aff2-56de3a2071a8-kube-api-access-4lr8d\") pod \"authorino-6f6944bcdd-lt4xd\" (UID: \"925666af-27fd-41b8-aff2-56de3a2071a8\") " pod="kuadrant-system/authorino-6f6944bcdd-lt4xd" Nov 03 20:55:31.392347 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:31.392315 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lt4xd" Nov 03 20:55:31.461924 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:31.461867 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lt4xd"] Nov 03 20:55:31.485453 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:31.485422 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b5c4d8cc8-zmf7q"] Nov 03 20:55:31.488532 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:31.488509 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b5c4d8cc8-zmf7q" Nov 03 20:55:31.542092 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:31.542061 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b5c4d8cc8-zmf7q"] Nov 03 20:55:31.616994 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:31.616956 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4wrs\" (UniqueName: \"kubernetes.io/projected/3ec30b41-2da4-48ff-9c77-41710291efa2-kube-api-access-x4wrs\") pod \"authorino-7b5c4d8cc8-zmf7q\" (UID: \"3ec30b41-2da4-48ff-9c77-41710291efa2\") " pod="kuadrant-system/authorino-7b5c4d8cc8-zmf7q" Nov 03 20:55:31.718175 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:31.718132 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4wrs\" (UniqueName: \"kubernetes.io/projected/3ec30b41-2da4-48ff-9c77-41710291efa2-kube-api-access-x4wrs\") pod \"authorino-7b5c4d8cc8-zmf7q\" (UID: \"3ec30b41-2da4-48ff-9c77-41710291efa2\") " pod="kuadrant-system/authorino-7b5c4d8cc8-zmf7q" Nov 03 20:55:31.759570 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:31.759533 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lt4xd"] Nov 03 20:55:31.765646 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:55:31.765614 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod925666af_27fd_41b8_aff2_56de3a2071a8.slice/crio-a1a1286a9bedcd1cc769f227ac659cc41c0203a3eea264203ca32464ea453384 WatchSource:0}: Error finding container a1a1286a9bedcd1cc769f227ac659cc41c0203a3eea264203ca32464ea453384: Status 404 returned error can't find the container with id a1a1286a9bedcd1cc769f227ac659cc41c0203a3eea264203ca32464ea453384 Nov 03 20:55:31.777670 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:31.777640 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4wrs\" (UniqueName: \"kubernetes.io/projected/3ec30b41-2da4-48ff-9c77-41710291efa2-kube-api-access-x4wrs\") pod \"authorino-7b5c4d8cc8-zmf7q\" (UID: \"3ec30b41-2da4-48ff-9c77-41710291efa2\") " pod="kuadrant-system/authorino-7b5c4d8cc8-zmf7q" Nov 03 20:55:31.786869 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:31.786842 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b5c4d8cc8-zmf7q"] Nov 03 20:55:31.787135 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:31.787119 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b5c4d8cc8-zmf7q" Nov 03 20:55:32.156712 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:32.156623 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b5c4d8cc8-zmf7q"] Nov 03 20:55:32.163224 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:55:32.163184 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ec30b41_2da4_48ff_9c77_41710291efa2.slice/crio-2aa358b2247d81cb062d8b1c248f42f975b8b4522c3e1c543bfe44bb3652e0bc WatchSource:0}: Error finding container 2aa358b2247d81cb062d8b1c248f42f975b8b4522c3e1c543bfe44bb3652e0bc: Status 404 returned error can't find the container with id 2aa358b2247d81cb062d8b1c248f42f975b8b4522c3e1c543bfe44bb3652e0bc Nov 03 20:55:32.277223 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:32.277184 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lt4xd" event={"ID":"925666af-27fd-41b8-aff2-56de3a2071a8","Type":"ContainerStarted","Data":"a1a1286a9bedcd1cc769f227ac659cc41c0203a3eea264203ca32464ea453384"} Nov 03 20:55:32.278447 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:32.278420 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b5c4d8cc8-zmf7q" event={"ID":"3ec30b41-2da4-48ff-9c77-41710291efa2","Type":"ContainerStarted","Data":"2aa358b2247d81cb062d8b1c248f42f975b8b4522c3e1c543bfe44bb3652e0bc"} Nov 03 20:55:34.291545 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:34.291506 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lt4xd" event={"ID":"925666af-27fd-41b8-aff2-56de3a2071a8","Type":"ContainerStarted","Data":"7910cd2f4c0e5c2e2a8967b253dfdf11afa1bedbdfb53fc28b7d712ce575dc0e"} Nov 03 20:55:34.292079 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:34.291563 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-lt4xd" podUID="925666af-27fd-41b8-aff2-56de3a2071a8" containerName="authorino" containerID="cri-o://7910cd2f4c0e5c2e2a8967b253dfdf11afa1bedbdfb53fc28b7d712ce575dc0e" gracePeriod=30 Nov 03 20:55:34.293451 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:34.293415 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b5c4d8cc8-zmf7q" event={"ID":"3ec30b41-2da4-48ff-9c77-41710291efa2","Type":"ContainerStarted","Data":"1d155fe6bd4a7e2ca6f15a077b393ada4ca50b928a26ebe1b3408788e32b55c6"} Nov 03 20:55:34.293587 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:34.293554 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7b5c4d8cc8-zmf7q" podUID="3ec30b41-2da4-48ff-9c77-41710291efa2" containerName="authorino" containerID="cri-o://1d155fe6bd4a7e2ca6f15a077b393ada4ca50b928a26ebe1b3408788e32b55c6" gracePeriod=30 Nov 03 20:55:34.401412 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:34.401348 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-lt4xd" podStartSLOduration=1.743264206 podStartE2EDuration="3.401334615s" totalImagesPullingTime="1.658070409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:34.399156961 +0000 UTC m=+483.559229235" watchObservedRunningTime="2026-11-03 20:55:34.401334615 +0000 UTC m=+483.561406886" Nov 03 20:55:34.508937 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:34.508856 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7b5c4d8cc8-zmf7q" podStartSLOduration=2.116701972 podStartE2EDuration="3.508832669s" totalImagesPullingTime="1.392130697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:34.506751114 +0000 UTC m=+483.666823387" watchObservedRunningTime="2026-11-03 20:55:34.508832669 +0000 UTC m=+483.668904942" Nov 03 20:55:34.570573 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:34.570546 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b5c4d8cc8-zmf7q" Nov 03 20:55:34.602720 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:34.602687 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lt4xd" Nov 03 20:55:34.643313 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:34.643278 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec30b41-2da4-48ff-9c77-41710291efa2-kube-api-access-x4wrs\" (UniqueName: \"kubernetes.io/projected/3ec30b41-2da4-48ff-9c77-41710291efa2-kube-api-access-x4wrs\") pod \"3ec30b41-2da4-48ff-9c77-41710291efa2\" (UID: \"3ec30b41-2da4-48ff-9c77-41710291efa2\") " Nov 03 20:55:34.643487 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:34.643329 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925666af-27fd-41b8-aff2-56de3a2071a8-kube-api-access-4lr8d\" (UniqueName: \"kubernetes.io/projected/925666af-27fd-41b8-aff2-56de3a2071a8-kube-api-access-4lr8d\") pod \"925666af-27fd-41b8-aff2-56de3a2071a8\" (UID: \"925666af-27fd-41b8-aff2-56de3a2071a8\") " Nov 03 20:55:34.645506 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:34.645475 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec30b41-2da4-48ff-9c77-41710291efa2-kube-api-access-x4wrs" pod "3ec30b41-2da4-48ff-9c77-41710291efa2" (UID: "3ec30b41-2da4-48ff-9c77-41710291efa2"). InnerVolumeSpecName "kube-api-access-x4wrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:34.645624 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:34.645503 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925666af-27fd-41b8-aff2-56de3a2071a8-kube-api-access-4lr8d" pod "925666af-27fd-41b8-aff2-56de3a2071a8" (UID: "925666af-27fd-41b8-aff2-56de3a2071a8"). InnerVolumeSpecName "kube-api-access-4lr8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:34.744694 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:34.744663 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4wrs\" (UniqueName: \"kubernetes.io/projected/3ec30b41-2da4-48ff-9c77-41710291efa2-kube-api-access-x4wrs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:34.744694 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:34.744691 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lr8d\" (UniqueName: \"kubernetes.io/projected/925666af-27fd-41b8-aff2-56de3a2071a8-kube-api-access-4lr8d\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:35.299390 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:35.299290 4963 generic.go:356] "Generic (PLEG): container finished" podID="925666af-27fd-41b8-aff2-56de3a2071a8" containerID="7910cd2f4c0e5c2e2a8967b253dfdf11afa1bedbdfb53fc28b7d712ce575dc0e" exitCode=0 Nov 03 20:55:35.299390 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:35.299339 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lt4xd" Nov 03 20:55:35.299390 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:35.299373 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lt4xd" event={"ID":"925666af-27fd-41b8-aff2-56de3a2071a8","Type":"ContainerDied","Data":"7910cd2f4c0e5c2e2a8967b253dfdf11afa1bedbdfb53fc28b7d712ce575dc0e"} Nov 03 20:55:35.299972 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:35.299418 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lt4xd" event={"ID":"925666af-27fd-41b8-aff2-56de3a2071a8","Type":"ContainerDied","Data":"a1a1286a9bedcd1cc769f227ac659cc41c0203a3eea264203ca32464ea453384"} Nov 03 20:55:35.299972 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:35.299446 4963 scope.go:122] "RemoveContainer" containerID="7910cd2f4c0e5c2e2a8967b253dfdf11afa1bedbdfb53fc28b7d712ce575dc0e" Nov 03 20:55:35.300825 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:35.300796 4963 generic.go:356] "Generic (PLEG): container finished" podID="3ec30b41-2da4-48ff-9c77-41710291efa2" containerID="1d155fe6bd4a7e2ca6f15a077b393ada4ca50b928a26ebe1b3408788e32b55c6" exitCode=0 Nov 03 20:55:35.301021 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:35.300838 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b5c4d8cc8-zmf7q" Nov 03 20:55:35.301021 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:35.300847 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b5c4d8cc8-zmf7q" event={"ID":"3ec30b41-2da4-48ff-9c77-41710291efa2","Type":"ContainerDied","Data":"1d155fe6bd4a7e2ca6f15a077b393ada4ca50b928a26ebe1b3408788e32b55c6"} Nov 03 20:55:35.301021 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:35.300890 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b5c4d8cc8-zmf7q" event={"ID":"3ec30b41-2da4-48ff-9c77-41710291efa2","Type":"ContainerDied","Data":"2aa358b2247d81cb062d8b1c248f42f975b8b4522c3e1c543bfe44bb3652e0bc"} Nov 03 20:55:35.308501 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:35.308479 4963 scope.go:122] "RemoveContainer" containerID="7910cd2f4c0e5c2e2a8967b253dfdf11afa1bedbdfb53fc28b7d712ce575dc0e" Nov 03 20:55:35.308820 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:55:35.308798 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7910cd2f4c0e5c2e2a8967b253dfdf11afa1bedbdfb53fc28b7d712ce575dc0e\": container with ID starting with 7910cd2f4c0e5c2e2a8967b253dfdf11afa1bedbdfb53fc28b7d712ce575dc0e not found: ID does not exist" containerID="7910cd2f4c0e5c2e2a8967b253dfdf11afa1bedbdfb53fc28b7d712ce575dc0e" Nov 03 20:55:35.308929 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:35.308830 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7910cd2f4c0e5c2e2a8967b253dfdf11afa1bedbdfb53fc28b7d712ce575dc0e"} err="failed to get container status \"7910cd2f4c0e5c2e2a8967b253dfdf11afa1bedbdfb53fc28b7d712ce575dc0e\": rpc error: code = NotFound desc = could not find container \"7910cd2f4c0e5c2e2a8967b253dfdf11afa1bedbdfb53fc28b7d712ce575dc0e\": container with ID starting with 7910cd2f4c0e5c2e2a8967b253dfdf11afa1bedbdfb53fc28b7d712ce575dc0e not found: ID does not exist" Nov 03 20:55:35.308929 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:35.308848 4963 scope.go:122] "RemoveContainer" containerID="1d155fe6bd4a7e2ca6f15a077b393ada4ca50b928a26ebe1b3408788e32b55c6" Nov 03 20:55:35.317581 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:35.317562 4963 scope.go:122] "RemoveContainer" containerID="1d155fe6bd4a7e2ca6f15a077b393ada4ca50b928a26ebe1b3408788e32b55c6" Nov 03 20:55:35.317920 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:55:35.317887 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d155fe6bd4a7e2ca6f15a077b393ada4ca50b928a26ebe1b3408788e32b55c6\": container with ID starting with 1d155fe6bd4a7e2ca6f15a077b393ada4ca50b928a26ebe1b3408788e32b55c6 not found: ID does not exist" containerID="1d155fe6bd4a7e2ca6f15a077b393ada4ca50b928a26ebe1b3408788e32b55c6" Nov 03 20:55:35.317995 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:35.317941 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d155fe6bd4a7e2ca6f15a077b393ada4ca50b928a26ebe1b3408788e32b55c6"} err="failed to get container status \"1d155fe6bd4a7e2ca6f15a077b393ada4ca50b928a26ebe1b3408788e32b55c6\": rpc error: code = NotFound desc = could not find container \"1d155fe6bd4a7e2ca6f15a077b393ada4ca50b928a26ebe1b3408788e32b55c6\": container with ID starting with 1d155fe6bd4a7e2ca6f15a077b393ada4ca50b928a26ebe1b3408788e32b55c6 not found: ID does not exist" Nov 03 20:55:35.454644 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:35.454605 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lt4xd"] Nov 03 20:55:35.458660 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:35.458623 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lt4xd"] Nov 03 20:55:35.565512 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:35.565422 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b5c4d8cc8-zmf7q"] Nov 03 20:55:35.570344 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:35.570315 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7b5c4d8cc8-zmf7q"] Nov 03 20:55:36.970032 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:36.969996 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ec30b41-2da4-48ff-9c77-41710291efa2" path="/var/lib/kubelet/pods/3ec30b41-2da4-48ff-9c77-41710291efa2/volumes" Nov 03 20:55:36.970549 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:36.970499 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="925666af-27fd-41b8-aff2-56de3a2071a8" path="/var/lib/kubelet/pods/925666af-27fd-41b8-aff2-56de3a2071a8/volumes" Nov 03 20:55:39.162366 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:39.162332 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n" Nov 03 20:55:42.528220 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:55:42.528134 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:55:44.559437 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:44.559403 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw"] Nov 03 20:55:44.559917 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:44.559677 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" podUID="699b9513-7af2-4d80-9f67-105257832239" containerName="maas-api" containerID="cri-o://b424a7a49b14323266671ea23b62d77beb9b530a46637424ad27e0d109d86612" gracePeriod=30 Nov 03 20:55:44.817633 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:44.817560 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" Nov 03 20:55:44.936875 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:44.936838 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/699b9513-7af2-4d80-9f67-105257832239-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699b9513-7af2-4d80-9f67-105257832239-maas-api-tls\") pod \"699b9513-7af2-4d80-9f67-105257832239\" (UID: \"699b9513-7af2-4d80-9f67-105257832239\") " Nov 03 20:55:44.937123 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:44.936886 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/699b9513-7af2-4d80-9f67-105257832239-kube-api-access-nm9qg\" (UniqueName: \"kubernetes.io/projected/699b9513-7af2-4d80-9f67-105257832239-kube-api-access-nm9qg\") pod \"699b9513-7af2-4d80-9f67-105257832239\" (UID: \"699b9513-7af2-4d80-9f67-105257832239\") " Nov 03 20:55:44.937123 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:44.937035 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/699b9513-7af2-4d80-9f67-105257832239-metrics-certs\" (UniqueName: \"kubernetes.io/secret/699b9513-7af2-4d80-9f67-105257832239-metrics-certs\") pod \"699b9513-7af2-4d80-9f67-105257832239\" (UID: \"699b9513-7af2-4d80-9f67-105257832239\") " Nov 03 20:55:44.937260 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:44.937190 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/699b9513-7af2-4d80-9f67-105257832239-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699b9513-7af2-4d80-9f67-105257832239-openshift-service-ca\") pod \"699b9513-7af2-4d80-9f67-105257832239\" (UID: \"699b9513-7af2-4d80-9f67-105257832239\") " Nov 03 20:55:44.937809 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:44.937778 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699b9513-7af2-4d80-9f67-105257832239-openshift-service-ca" pod "699b9513-7af2-4d80-9f67-105257832239" (UID: "699b9513-7af2-4d80-9f67-105257832239"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:44.939307 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:44.939272 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699b9513-7af2-4d80-9f67-105257832239-kube-api-access-nm9qg" pod "699b9513-7af2-4d80-9f67-105257832239" (UID: "699b9513-7af2-4d80-9f67-105257832239"). InnerVolumeSpecName "kube-api-access-nm9qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:44.939307 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:44.939294 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/699b9513-7af2-4d80-9f67-105257832239-metrics-certs" pod "699b9513-7af2-4d80-9f67-105257832239" (UID: "699b9513-7af2-4d80-9f67-105257832239"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:44.939491 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:44.939343 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/699b9513-7af2-4d80-9f67-105257832239-maas-api-tls" pod "699b9513-7af2-4d80-9f67-105257832239" (UID: "699b9513-7af2-4d80-9f67-105257832239"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:45.038099 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:45.038063 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699b9513-7af2-4d80-9f67-105257832239-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:45.038099 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:45.038092 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699b9513-7af2-4d80-9f67-105257832239-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:45.038099 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:45.038102 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm9qg\" (UniqueName: \"kubernetes.io/projected/699b9513-7af2-4d80-9f67-105257832239-kube-api-access-nm9qg\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:45.038337 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:45.038112 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/699b9513-7af2-4d80-9f67-105257832239-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:45.354493 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:45.354456 4963 generic.go:356] "Generic (PLEG): container finished" podID="699b9513-7af2-4d80-9f67-105257832239" containerID="b424a7a49b14323266671ea23b62d77beb9b530a46637424ad27e0d109d86612" exitCode=0 Nov 03 20:55:45.354669 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:45.354504 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" Nov 03 20:55:45.354669 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:45.354541 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" event={"ID":"699b9513-7af2-4d80-9f67-105257832239","Type":"ContainerDied","Data":"b424a7a49b14323266671ea23b62d77beb9b530a46637424ad27e0d109d86612"} Nov 03 20:55:45.354669 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:45.354597 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw" event={"ID":"699b9513-7af2-4d80-9f67-105257832239","Type":"ContainerDied","Data":"d49b1fd70040458d3671a8a1d96b846516aa428c037b10188c39358c33b40851"} Nov 03 20:55:45.354669 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:45.354625 4963 scope.go:122] "RemoveContainer" containerID="b424a7a49b14323266671ea23b62d77beb9b530a46637424ad27e0d109d86612" Nov 03 20:55:45.363587 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:45.363562 4963 scope.go:122] "RemoveContainer" containerID="b424a7a49b14323266671ea23b62d77beb9b530a46637424ad27e0d109d86612" Nov 03 20:55:45.363892 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:55:45.363870 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b424a7a49b14323266671ea23b62d77beb9b530a46637424ad27e0d109d86612\": container with ID starting with b424a7a49b14323266671ea23b62d77beb9b530a46637424ad27e0d109d86612 not found: ID does not exist" containerID="b424a7a49b14323266671ea23b62d77beb9b530a46637424ad27e0d109d86612" Nov 03 20:55:45.364018 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:45.363928 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b424a7a49b14323266671ea23b62d77beb9b530a46637424ad27e0d109d86612"} err="failed to get container status \"b424a7a49b14323266671ea23b62d77beb9b530a46637424ad27e0d109d86612\": rpc error: code = NotFound desc = could not find container \"b424a7a49b14323266671ea23b62d77beb9b530a46637424ad27e0d109d86612\": container with ID starting with b424a7a49b14323266671ea23b62d77beb9b530a46637424ad27e0d109d86612 not found: ID does not exist" Nov 03 20:55:45.510679 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:45.510640 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw"] Nov 03 20:55:45.557932 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:45.557875 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dfd88bddd-xgckw"] Nov 03 20:55:46.965894 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:55:46.965859 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="699b9513-7af2-4d80-9f67-105257832239" path="/var/lib/kubelet/pods/699b9513-7af2-4d80-9f67-105257832239/volumes" Nov 03 20:56:12.527726 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:56:12.527691 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:56:15.889868 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:15.889832 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-6m794"] Nov 03 20:56:15.890429 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:15.890409 4963 cpu_manager.go:414] "containerMap: removing container" podUID="699b9513-7af2-4d80-9f67-105257832239" containerName="maas-api" Nov 03 20:56:15.890517 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:15.890432 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="699b9513-7af2-4d80-9f67-105257832239" containerName="maas-api" Nov 03 20:56:15.890517 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:15.890475 4963 cpu_manager.go:414] "containerMap: removing container" podUID="925666af-27fd-41b8-aff2-56de3a2071a8" containerName="authorino" Nov 03 20:56:15.890517 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:15.890484 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="925666af-27fd-41b8-aff2-56de3a2071a8" containerName="authorino" Nov 03 20:56:15.890517 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:15.890507 4963 cpu_manager.go:414] "containerMap: removing container" podUID="3ec30b41-2da4-48ff-9c77-41710291efa2" containerName="authorino" Nov 03 20:56:15.890517 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:15.890514 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec30b41-2da4-48ff-9c77-41710291efa2" containerName="authorino" Nov 03 20:56:15.890751 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:15.890622 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec30b41-2da4-48ff-9c77-41710291efa2" containerName="authorino" Nov 03 20:56:15.890751 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:15.890640 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="699b9513-7af2-4d80-9f67-105257832239" containerName="maas-api" Nov 03 20:56:15.890751 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:15.890654 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="925666af-27fd-41b8-aff2-56de3a2071a8" containerName="authorino" Nov 03 20:56:15.893043 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:15.893022 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6m794" Nov 03 20:56:15.948722 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:15.948687 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-6m794"] Nov 03 20:56:16.018608 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:16.018568 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24076c9b-c504-4b0c-82dc-d95b44f0e977-cert\") pod \"llmisvc-controller-manager-6466c554d8-6m794\" (UID: \"24076c9b-c504-4b0c-82dc-d95b44f0e977\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6m794" Nov 03 20:56:16.018608 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:16.018610 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq5kb\" (UniqueName: \"kubernetes.io/projected/24076c9b-c504-4b0c-82dc-d95b44f0e977-kube-api-access-nq5kb\") pod \"llmisvc-controller-manager-6466c554d8-6m794\" (UID: \"24076c9b-c504-4b0c-82dc-d95b44f0e977\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6m794" Nov 03 20:56:16.120133 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:16.120092 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24076c9b-c504-4b0c-82dc-d95b44f0e977-cert\") pod \"llmisvc-controller-manager-6466c554d8-6m794\" (UID: \"24076c9b-c504-4b0c-82dc-d95b44f0e977\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6m794" Nov 03 20:56:16.120133 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:16.120129 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq5kb\" (UniqueName: \"kubernetes.io/projected/24076c9b-c504-4b0c-82dc-d95b44f0e977-kube-api-access-nq5kb\") pod \"llmisvc-controller-manager-6466c554d8-6m794\" (UID: \"24076c9b-c504-4b0c-82dc-d95b44f0e977\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6m794" Nov 03 20:56:16.122608 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:16.122583 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24076c9b-c504-4b0c-82dc-d95b44f0e977-cert\") pod \"llmisvc-controller-manager-6466c554d8-6m794\" (UID: \"24076c9b-c504-4b0c-82dc-d95b44f0e977\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6m794" Nov 03 20:56:16.178876 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:16.178808 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq5kb\" (UniqueName: \"kubernetes.io/projected/24076c9b-c504-4b0c-82dc-d95b44f0e977-kube-api-access-nq5kb\") pod \"llmisvc-controller-manager-6466c554d8-6m794\" (UID: \"24076c9b-c504-4b0c-82dc-d95b44f0e977\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6m794" Nov 03 20:56:16.204656 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:16.204614 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6m794" Nov 03 20:56:16.573424 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:16.573389 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-6m794"] Nov 03 20:56:16.579842 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:56:16.579812 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24076c9b_c504_4b0c_82dc_d95b44f0e977.slice/crio-82e220426ac2343c06ac5a01ba36b076619a6b41e97c6b5e09d78bf8fdc214c1 WatchSource:0}: Error finding container 82e220426ac2343c06ac5a01ba36b076619a6b41e97c6b5e09d78bf8fdc214c1: Status 404 returned error can't find the container with id 82e220426ac2343c06ac5a01ba36b076619a6b41e97c6b5e09d78bf8fdc214c1 Nov 03 20:56:17.507967 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:17.507896 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6m794" event={"ID":"24076c9b-c504-4b0c-82dc-d95b44f0e977","Type":"ContainerStarted","Data":"b1bf6cad785f3e8c35cd35b15913f0a25ae17234838b1b6551fc7fd9272a9528"} Nov 03 20:56:17.508578 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:17.507998 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6m794" Nov 03 20:56:17.508578 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:17.508022 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6m794" event={"ID":"24076c9b-c504-4b0c-82dc-d95b44f0e977","Type":"ContainerStarted","Data":"82e220426ac2343c06ac5a01ba36b076619a6b41e97c6b5e09d78bf8fdc214c1"} Nov 03 20:56:17.618706 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:17.618650 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6m794" podStartSLOduration=1.806702601 podStartE2EDuration="2.618634117s" totalImagesPullingTime="811.931516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:17.617538677 +0000 UTC m=+526.777610949" watchObservedRunningTime="2026-11-03 20:56:17.618634117 +0000 UTC m=+526.778706389" Nov 03 20:56:24.033571 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:24.033532 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:42.527207 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:56:42.527163 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:56:48.515989 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:48.515954 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6m794" Nov 03 20:56:48.676051 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:48.676009 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n"] Nov 03 20:56:48.676313 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:48.676269 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n" podUID="26c4944c-af22-41af-8270-ad1e85513dab" containerName="manager" containerID="cri-o://37ba629e890a073ed2586dd42e313e2610f3570d389984a51e4b70cfda1c1de1" gracePeriod=30 Nov 03 20:56:48.936785 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:48.936759 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n" Nov 03 20:56:49.005497 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:49.005462 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c4944c-af22-41af-8270-ad1e85513dab-kube-api-access-n7lll\" (UniqueName: \"kubernetes.io/projected/26c4944c-af22-41af-8270-ad1e85513dab-kube-api-access-n7lll\") pod \"26c4944c-af22-41af-8270-ad1e85513dab\" (UID: \"26c4944c-af22-41af-8270-ad1e85513dab\") " Nov 03 20:56:49.005664 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:49.005518 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26c4944c-af22-41af-8270-ad1e85513dab-cert\" (UniqueName: \"kubernetes.io/secret/26c4944c-af22-41af-8270-ad1e85513dab-cert\") pod \"26c4944c-af22-41af-8270-ad1e85513dab\" (UID: \"26c4944c-af22-41af-8270-ad1e85513dab\") " Nov 03 20:56:49.007724 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:49.007697 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c4944c-af22-41af-8270-ad1e85513dab-cert" pod "26c4944c-af22-41af-8270-ad1e85513dab" (UID: "26c4944c-af22-41af-8270-ad1e85513dab"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:49.007841 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:49.007744 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c4944c-af22-41af-8270-ad1e85513dab-kube-api-access-n7lll" pod "26c4944c-af22-41af-8270-ad1e85513dab" (UID: "26c4944c-af22-41af-8270-ad1e85513dab"). InnerVolumeSpecName "kube-api-access-n7lll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:49.107239 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:49.107149 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7lll\" (UniqueName: \"kubernetes.io/projected/26c4944c-af22-41af-8270-ad1e85513dab-kube-api-access-n7lll\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:49.107239 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:49.107182 4963 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26c4944c-af22-41af-8270-ad1e85513dab-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:49.664512 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:49.664476 4963 generic.go:356] "Generic (PLEG): container finished" podID="26c4944c-af22-41af-8270-ad1e85513dab" containerID="37ba629e890a073ed2586dd42e313e2610f3570d389984a51e4b70cfda1c1de1" exitCode=0 Nov 03 20:56:49.665038 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:49.664522 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n" Nov 03 20:56:49.665038 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:49.664564 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n" event={"ID":"26c4944c-af22-41af-8270-ad1e85513dab","Type":"ContainerDied","Data":"37ba629e890a073ed2586dd42e313e2610f3570d389984a51e4b70cfda1c1de1"} Nov 03 20:56:49.665038 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:49.664624 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n" event={"ID":"26c4944c-af22-41af-8270-ad1e85513dab","Type":"ContainerDied","Data":"52f9296b1356c4475becb7d9d0a1b72933f0582df31b69f64dd7e2495d85fb71"} Nov 03 20:56:49.665038 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:49.664658 4963 scope.go:122] "RemoveContainer" containerID="37ba629e890a073ed2586dd42e313e2610f3570d389984a51e4b70cfda1c1de1" Nov 03 20:56:49.676370 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:49.676036 4963 scope.go:122] "RemoveContainer" containerID="37ba629e890a073ed2586dd42e313e2610f3570d389984a51e4b70cfda1c1de1" Nov 03 20:56:49.677145 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:56:49.677116 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ba629e890a073ed2586dd42e313e2610f3570d389984a51e4b70cfda1c1de1\": container with ID starting with 37ba629e890a073ed2586dd42e313e2610f3570d389984a51e4b70cfda1c1de1 not found: ID does not exist" containerID="37ba629e890a073ed2586dd42e313e2610f3570d389984a51e4b70cfda1c1de1" Nov 03 20:56:49.677263 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:49.677155 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ba629e890a073ed2586dd42e313e2610f3570d389984a51e4b70cfda1c1de1"} err="failed to get container status \"37ba629e890a073ed2586dd42e313e2610f3570d389984a51e4b70cfda1c1de1\": rpc error: code = NotFound desc = could not find container \"37ba629e890a073ed2586dd42e313e2610f3570d389984a51e4b70cfda1c1de1\": container with ID starting with 37ba629e890a073ed2586dd42e313e2610f3570d389984a51e4b70cfda1c1de1 not found: ID does not exist" Nov 03 20:56:49.821702 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:49.821672 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n"] Nov 03 20:56:49.868978 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:49.868943 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5wz4n"] Nov 03 20:56:50.967808 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:50.967769 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c4944c-af22-41af-8270-ad1e85513dab" path="/var/lib/kubelet/pods/26c4944c-af22-41af-8270-ad1e85513dab/volumes" Nov 03 20:56:51.963605 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:51.963562 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:52.235636 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:56:52.235548 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:00.813254 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:57:00.813218 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:00.999153 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:57:00.999115 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:12.528092 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:57:12.527961 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:57:24.331594 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:57:24.331551 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:24.683775 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:57:24.683678 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:25.209178 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:57:25.209139 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:26.478348 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:57:26.478301 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:26.664856 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:57:26.664825 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:31.008144 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:57:31.008110 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 20:57:31.010402 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:57:31.010377 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 20:57:33.262342 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:57:33.262307 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:39.505735 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:57:39.505699 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:42.528366 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:57:42.528327 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:57:43.195446 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:57:43.195404 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:46.985362 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:57:46.985327 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:47.970120 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:57:47.970083 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:48.152297 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:57:48.152261 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:50.504349 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:57:50.504303 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:53.893537 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:57:53.893503 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:59.833399 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:57:59.833364 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:03.820448 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:03.820411 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:12.526273 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:58:12.526229 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:58:24.671246 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:24.671206 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:26.138352 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.138320 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb"] Nov 03 20:58:26.138940 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.138918 4963 cpu_manager.go:414] "containerMap: removing container" podUID="26c4944c-af22-41af-8270-ad1e85513dab" containerName="manager" Nov 03 20:58:26.139034 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.138942 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c4944c-af22-41af-8270-ad1e85513dab" containerName="manager" Nov 03 20:58:26.139100 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.139042 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c4944c-af22-41af-8270-ad1e85513dab" containerName="manager" Nov 03 20:58:26.142186 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.142164 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.167276 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.167242 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.167276 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.167278 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.167462 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.167298 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.167462 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.167418 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.167462 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.167444 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81c8de6a-8867-40ba-b88c-cb0d73475112-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.167558 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.167525 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.167594 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.167567 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.167638 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.167624 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsjpx\" (UniqueName: \"kubernetes.io/projected/81c8de6a-8867-40ba-b88c-cb0d73475112-kube-api-access-dsjpx\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.167673 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.167649 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.192220 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.192182 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4cjrg\"" Nov 03 20:58:26.241960 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.241927 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb"] Nov 03 20:58:26.269098 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.269061 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.269291 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.269128 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsjpx\" (UniqueName: \"kubernetes.io/projected/81c8de6a-8867-40ba-b88c-cb0d73475112-kube-api-access-dsjpx\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.269291 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.269161 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.269291 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.269203 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.269291 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.269237 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.269530 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.269396 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.269530 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.269475 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.269530 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.269502 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81c8de6a-8867-40ba-b88c-cb0d73475112-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.269717 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.269523 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.269717 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.269548 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.269842 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.269710 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.270220 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.269985 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.270220 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.270163 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.270384 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.270301 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81c8de6a-8867-40ba-b88c-cb0d73475112-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.272108 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.272082 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.272315 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.272294 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.327708 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.327669 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.327854 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.327811 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsjpx\" (UniqueName: \"kubernetes.io/projected/81c8de6a-8867-40ba-b88c-cb0d73475112-kube-api-access-dsjpx\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-rl6tb\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.455045 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.455006 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:26.879174 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.879139 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb"] Nov 03 20:58:26.885336 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:58:26.885304 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c8de6a_8867_40ba_b88c_cb0d73475112.slice/crio-b2327e5256359edc7cafb0f94361e3a4f4eca32985e26c24bff298f62a6877cb WatchSource:0}: Error finding container b2327e5256359edc7cafb0f94361e3a4f4eca32985e26c24bff298f62a6877cb: Status 404 returned error can't find the container with id b2327e5256359edc7cafb0f94361e3a4f4eca32985e26c24bff298f62a6877cb Nov 03 20:58:26.887426 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.887396 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:58:26.887515 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.887458 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:58:26.887515 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:26.887486 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:58:27.141010 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:27.140925 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" event={"ID":"81c8de6a-8867-40ba-b88c-cb0d73475112","Type":"ContainerStarted","Data":"60f60e8db8dbca66114b5976a30eef8f6507c890fa9408fdf8c8d41fd21444d7"} Nov 03 20:58:27.141010 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:27.140986 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" event={"ID":"81c8de6a-8867-40ba-b88c-cb0d73475112","Type":"ContainerStarted","Data":"b2327e5256359edc7cafb0f94361e3a4f4eca32985e26c24bff298f62a6877cb"} Nov 03 20:58:27.256731 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:27.256678 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" podStartSLOduration=1.256664365 podStartE2EDuration="1.256664365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:27.255823366 +0000 UTC m=+656.415895639" watchObservedRunningTime="2026-11-03 20:58:27.256664365 +0000 UTC m=+656.416736636" Nov 03 20:58:27.455578 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:27.455546 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:27.457094 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:27.457069 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:58:27.457234 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:27.457122 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" podUID="81c8de6a-8867-40ba-b88c-cb0d73475112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 20:58:28.455768 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:28.455729 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:58:28.456204 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:28.455787 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" podUID="81c8de6a-8867-40ba-b88c-cb0d73475112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 20:58:29.456291 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:29.456255 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:58:29.456679 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:29.456308 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" podUID="81c8de6a-8867-40ba-b88c-cb0d73475112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 20:58:30.455838 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:30.455797 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:58:30.456160 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:30.455856 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" podUID="81c8de6a-8867-40ba-b88c-cb0d73475112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 20:58:30.458898 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:30.458864 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-579685d67f-v4cx7"] Nov 03 20:58:30.462684 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:30.462663 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-579685d67f-v4cx7" Nov 03 20:58:30.529807 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:30.529772 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e175791-e600-4f34-b7f4-688619a990e1-tls-cert\") pod \"authorino-579685d67f-v4cx7\" (UID: \"7e175791-e600-4f34-b7f4-688619a990e1\") " pod="kuadrant-system/authorino-579685d67f-v4cx7" Nov 03 20:58:30.530011 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:30.529924 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjsxh\" (UniqueName: \"kubernetes.io/projected/7e175791-e600-4f34-b7f4-688619a990e1-kube-api-access-fjsxh\") pod \"authorino-579685d67f-v4cx7\" (UID: \"7e175791-e600-4f34-b7f4-688619a990e1\") " pod="kuadrant-system/authorino-579685d67f-v4cx7" Nov 03 20:58:30.557760 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:30.557716 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-579685d67f-v4cx7"] Nov 03 20:58:30.560547 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:30.560530 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 20:58:30.560547 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:30.560538 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dq8fb\"" Nov 03 20:58:30.630962 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:30.630890 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e175791-e600-4f34-b7f4-688619a990e1-tls-cert\") pod \"authorino-579685d67f-v4cx7\" (UID: \"7e175791-e600-4f34-b7f4-688619a990e1\") " pod="kuadrant-system/authorino-579685d67f-v4cx7" Nov 03 20:58:30.631139 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:30.631033 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjsxh\" (UniqueName: \"kubernetes.io/projected/7e175791-e600-4f34-b7f4-688619a990e1-kube-api-access-fjsxh\") pod \"authorino-579685d67f-v4cx7\" (UID: \"7e175791-e600-4f34-b7f4-688619a990e1\") " pod="kuadrant-system/authorino-579685d67f-v4cx7" Nov 03 20:58:30.633492 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:30.633460 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e175791-e600-4f34-b7f4-688619a990e1-tls-cert\") pod \"authorino-579685d67f-v4cx7\" (UID: \"7e175791-e600-4f34-b7f4-688619a990e1\") " pod="kuadrant-system/authorino-579685d67f-v4cx7" Nov 03 20:58:30.706690 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:30.706619 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjsxh\" (UniqueName: \"kubernetes.io/projected/7e175791-e600-4f34-b7f4-688619a990e1-kube-api-access-fjsxh\") pod \"authorino-579685d67f-v4cx7\" (UID: \"7e175791-e600-4f34-b7f4-688619a990e1\") " pod="kuadrant-system/authorino-579685d67f-v4cx7" Nov 03 20:58:30.774634 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:30.774596 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-579685d67f-v4cx7" Nov 03 20:58:31.168593 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:31.168514 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-579685d67f-v4cx7"] Nov 03 20:58:31.174835 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:58:31.174803 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e175791_e600_4f34_b7f4_688619a990e1.slice/crio-3d452a152d917eb0241cd3b67d6c5ad2308fccc5d3fc98682b129891be4dec2d WatchSource:0}: Error finding container 3d452a152d917eb0241cd3b67d6c5ad2308fccc5d3fc98682b129891be4dec2d: Status 404 returned error can't find the container with id 3d452a152d917eb0241cd3b67d6c5ad2308fccc5d3fc98682b129891be4dec2d Nov 03 20:58:31.176284 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:31.176263 4963 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:58:31.191928 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:31.190737 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:31.373532 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:31.373504 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:31.455410 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:31.455373 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:58:31.455587 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:31.455464 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" podUID="81c8de6a-8867-40ba-b88c-cb0d73475112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 20:58:32.168043 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:32.168007 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-579685d67f-v4cx7" event={"ID":"7e175791-e600-4f34-b7f4-688619a990e1","Type":"ContainerStarted","Data":"3d452a152d917eb0241cd3b67d6c5ad2308fccc5d3fc98682b129891be4dec2d"} Nov 03 20:58:32.456061 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:32.456025 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:58:32.456237 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:32.456102 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" podUID="81c8de6a-8867-40ba-b88c-cb0d73475112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 20:58:32.836832 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:32.836798 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:33.027516 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:33.027420 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:33.174998 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:33.174960 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-579685d67f-v4cx7" event={"ID":"7e175791-e600-4f34-b7f4-688619a990e1","Type":"ContainerStarted","Data":"567548793d7afed9c1c8a28870cdbb22b3296070060bdeab97609625ac7c972b"} Nov 03 20:58:33.292630 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:33.292511 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-579685d67f-v4cx7" podStartSLOduration=1.829852287 podStartE2EDuration="3.292490645s" totalImagesPullingTime="1.462638358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:33.292201488 +0000 UTC m=+662.452273760" watchObservedRunningTime="2026-11-03 20:58:33.292490645 +0000 UTC m=+662.452562918" Nov 03 20:58:33.455790 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:33.455747 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:58:33.456011 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:33.455798 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" podUID="81c8de6a-8867-40ba-b88c-cb0d73475112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 20:58:34.455763 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:34.455722 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:58:34.456270 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:34.455802 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" podUID="81c8de6a-8867-40ba-b88c-cb0d73475112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 20:58:35.496881 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:35.496842 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:35.497540 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:35.497516 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:35.498563 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:58:35.498540 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 20:58:42.527474 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:58:42.527389 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:59:03.687262 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:03.687227 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:03.880293 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:03.880262 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:04.074859 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:04.074824 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:04.270529 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:04.270490 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:04.600815 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:04.600775 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:04.787435 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:04.787395 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:04.973816 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:04.973778 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:05.158466 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:05.158438 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:05.544569 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:05.544535 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:05.756025 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:05.755992 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:05.946434 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:05.946400 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:07.105328 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:07.105294 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:07.420974 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:07.420864 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:07.623833 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:07.623801 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:07.813470 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:07.813436 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:08.602079 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:08.602047 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:08.790013 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:08.789979 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:09.672196 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:09.672163 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:10.536824 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:10.536789 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:10.730083 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:10.730047 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:10.919604 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:10.919516 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:11.097559 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:11.097526 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:11.547635 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:11.547594 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:11.771404 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:11.771369 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:11.954267 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:11.954233 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:12.528190 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:59:12.528154 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:59:12.631678 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:12.631641 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:12.874943 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:12.874822 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:13.374749 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:13.374713 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:13.716876 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:13.716842 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:13.904466 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:13.904434 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:14.087046 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:14.086945 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:14.271410 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:14.271372 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:15.764051 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:15.764018 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:15.947833 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:15.947801 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:17.448858 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:17.448813 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:18.179004 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.178970 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6"] Nov 03 20:59:18.185577 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.183480 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 20:59:18.238725 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.238689 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:59:18.238725 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.238699 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:59:18.269320 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.269276 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-openshift-service-ca\") pod \"maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6\" (UID: \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 20:59:18.269320 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.269313 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-maas-api-tls\") pod \"maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6\" (UID: \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 20:59:18.269591 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.269419 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-metrics-certs\") pod \"maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6\" (UID: \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 20:59:18.269591 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.269456 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv8q4\" (UniqueName: \"kubernetes.io/projected/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-kube-api-access-mv8q4\") pod \"maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6\" (UID: \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 20:59:18.286182 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.286152 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0a30f61a-serving-cert\"" Nov 03 20:59:18.286368 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.286199 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6whcf\"" Nov 03 20:59:18.286368 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.286324 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:59:18.288153 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.288133 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6"] Nov 03 20:59:18.294076 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.294052 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:18.370015 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.369978 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-metrics-certs\") pod \"maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6\" (UID: \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 20:59:18.370218 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.370035 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv8q4\" (UniqueName: \"kubernetes.io/projected/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-kube-api-access-mv8q4\") pod \"maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6\" (UID: \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 20:59:18.370218 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.370093 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-openshift-service-ca\") pod \"maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6\" (UID: \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 20:59:18.370218 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.370120 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-maas-api-tls\") pod \"maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6\" (UID: \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 20:59:18.370218 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:59:18.370211 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-serving-cert: secret "maas-api-e2e-ait-0a30f61a-serving-cert" not found Nov 03 20:59:18.370473 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:59:18.370285 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-maas-api-tls podName:0b193aba-8d0d-46f1-b62a-0c2b68c0f102 nodeName:}" failed. No retries permitted until 2026-11-03 20:59:18.870264995 +0000 UTC m=+708.030337251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-maas-api-tls") pod "maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" (UID: "0b193aba-8d0d-46f1-b62a-0c2b68c0f102") : secret "maas-api-e2e-ait-0a30f61a-serving-cert" not found Nov 03 20:59:18.371288 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.371259 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-openshift-service-ca\") pod \"maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6\" (UID: \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 20:59:18.372938 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.372897 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-metrics-certs\") pod \"maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6\" (UID: \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 20:59:18.430658 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.430591 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv8q4\" (UniqueName: \"kubernetes.io/projected/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-kube-api-access-mv8q4\") pod \"maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6\" (UID: \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 20:59:18.482183 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.482148 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:18.666387 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.666354 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:18.873498 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.873458 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-maas-api-tls\") pod \"maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6\" (UID: \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 20:59:18.876040 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:18.876018 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-maas-api-tls\") pod \"maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6\" (UID: \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 20:59:19.099269 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:19.099233 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 20:59:19.470658 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:19.470625 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6"] Nov 03 20:59:19.476819 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:59:19.476786 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b193aba_8d0d_46f1_b62a_0c2b68c0f102.slice/crio-650cbd4178c7a301ff12bcbd0f88841f9080b45f8e2508dff55d6e734f26b153 WatchSource:0}: Error finding container 650cbd4178c7a301ff12bcbd0f88841f9080b45f8e2508dff55d6e734f26b153: Status 404 returned error can't find the container with id 650cbd4178c7a301ff12bcbd0f88841f9080b45f8e2508dff55d6e734f26b153 Nov 03 20:59:19.561700 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:19.561660 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:19.758355 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:19.758269 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:19.946369 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:19.946330 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:20.414165 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:20.414125 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" event={"ID":"0b193aba-8d0d-46f1-b62a-0c2b68c0f102","Type":"ContainerStarted","Data":"650cbd4178c7a301ff12bcbd0f88841f9080b45f8e2508dff55d6e734f26b153"} Nov 03 20:59:20.895687 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:20.895656 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:21.074818 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:21.074784 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:21.421748 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:21.421712 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" event={"ID":"0b193aba-8d0d-46f1-b62a-0c2b68c0f102","Type":"ContainerStarted","Data":"a2376e46e95d3968c7424b6fafc5c1a04c343bae377cb325d7282356ef92a553"} Nov 03 20:59:21.421967 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:21.421876 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 20:59:21.534770 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:21.534699 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" podStartSLOduration=2.586637251 podStartE2EDuration="3.534683339s" totalImagesPullingTime="948.046088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:21.533316189 +0000 UTC m=+710.693388462" watchObservedRunningTime="2026-11-03 20:59:21.534683339 +0000 UTC m=+710.694755611" Nov 03 20:59:21.668777 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:21.668741 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:23.588810 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:23.588774 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:23.779756 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:23.779708 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:23.965487 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:23.965458 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:25.005708 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:25.005673 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:27.432399 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:27.432372 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 20:59:27.947263 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:27.947225 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm"] Nov 03 20:59:27.951104 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:27.951081 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 20:59:28.001161 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.001132 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-10bfccd7-serving-cert\"" Nov 03 20:59:28.006955 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.006921 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm"] Nov 03 20:59:28.044691 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.044659 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b87d4af-b1ba-4b14-a7f5-6260136e4151-metrics-certs\") pod \"maas-api-e2e-ait-10bfccd7-8467478557-42hcm\" (UID: \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 20:59:28.044865 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.044785 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bwgn\" (UniqueName: \"kubernetes.io/projected/0b87d4af-b1ba-4b14-a7f5-6260136e4151-kube-api-access-5bwgn\") pod \"maas-api-e2e-ait-10bfccd7-8467478557-42hcm\" (UID: \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 20:59:28.044865 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.044842 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b87d4af-b1ba-4b14-a7f5-6260136e4151-openshift-service-ca\") pod \"maas-api-e2e-ait-10bfccd7-8467478557-42hcm\" (UID: \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 20:59:28.044997 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.044897 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b87d4af-b1ba-4b14-a7f5-6260136e4151-maas-api-tls\") pod \"maas-api-e2e-ait-10bfccd7-8467478557-42hcm\" (UID: \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 20:59:28.145996 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.145966 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bwgn\" (UniqueName: \"kubernetes.io/projected/0b87d4af-b1ba-4b14-a7f5-6260136e4151-kube-api-access-5bwgn\") pod \"maas-api-e2e-ait-10bfccd7-8467478557-42hcm\" (UID: \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 20:59:28.146186 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.146011 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b87d4af-b1ba-4b14-a7f5-6260136e4151-openshift-service-ca\") pod \"maas-api-e2e-ait-10bfccd7-8467478557-42hcm\" (UID: \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 20:59:28.146186 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.146076 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b87d4af-b1ba-4b14-a7f5-6260136e4151-maas-api-tls\") pod \"maas-api-e2e-ait-10bfccd7-8467478557-42hcm\" (UID: \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 20:59:28.146186 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.146117 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b87d4af-b1ba-4b14-a7f5-6260136e4151-metrics-certs\") pod \"maas-api-e2e-ait-10bfccd7-8467478557-42hcm\" (UID: \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 20:59:28.146355 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:59:28.146221 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-serving-cert: secret "maas-api-e2e-ait-10bfccd7-serving-cert" not found Nov 03 20:59:28.146355 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:59:28.146296 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b87d4af-b1ba-4b14-a7f5-6260136e4151-maas-api-tls podName:0b87d4af-b1ba-4b14-a7f5-6260136e4151 nodeName:}" failed. No retries permitted until 2026-11-03 20:59:28.646280521 +0000 UTC m=+717.806352771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b87d4af-b1ba-4b14-a7f5-6260136e4151-maas-api-tls") pod "maas-api-e2e-ait-10bfccd7-8467478557-42hcm" (UID: "0b87d4af-b1ba-4b14-a7f5-6260136e4151") : secret "maas-api-e2e-ait-10bfccd7-serving-cert" not found Nov 03 20:59:28.146804 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.146781 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b87d4af-b1ba-4b14-a7f5-6260136e4151-openshift-service-ca\") pod \"maas-api-e2e-ait-10bfccd7-8467478557-42hcm\" (UID: \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 20:59:28.148589 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.148572 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b87d4af-b1ba-4b14-a7f5-6260136e4151-metrics-certs\") pod \"maas-api-e2e-ait-10bfccd7-8467478557-42hcm\" (UID: \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 20:59:28.182243 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.182211 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:28.207881 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.207812 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bwgn\" (UniqueName: \"kubernetes.io/projected/0b87d4af-b1ba-4b14-a7f5-6260136e4151-kube-api-access-5bwgn\") pod \"maas-api-e2e-ait-10bfccd7-8467478557-42hcm\" (UID: \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 20:59:28.363697 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.363658 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:28.548780 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.548690 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:28.651176 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.651141 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b87d4af-b1ba-4b14-a7f5-6260136e4151-maas-api-tls\") pod \"maas-api-e2e-ait-10bfccd7-8467478557-42hcm\" (UID: \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 20:59:28.653959 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.653925 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b87d4af-b1ba-4b14-a7f5-6260136e4151-maas-api-tls\") pod \"maas-api-e2e-ait-10bfccd7-8467478557-42hcm\" (UID: \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 20:59:28.864800 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:28.864706 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 20:59:29.237536 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:29.237498 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm"] Nov 03 20:59:29.243541 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:59:29.243505 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b87d4af_b1ba_4b14_a7f5_6260136e4151.slice/crio-d440bf12720238395d616f2ac17cdd4986783ac87d60382fb7ce69d5aba04112 WatchSource:0}: Error finding container d440bf12720238395d616f2ac17cdd4986783ac87d60382fb7ce69d5aba04112: Status 404 returned error can't find the container with id d440bf12720238395d616f2ac17cdd4986783ac87d60382fb7ce69d5aba04112 Nov 03 20:59:29.469378 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:29.469340 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" event={"ID":"0b87d4af-b1ba-4b14-a7f5-6260136e4151","Type":"ContainerStarted","Data":"d440bf12720238395d616f2ac17cdd4986783ac87d60382fb7ce69d5aba04112"} Nov 03 20:59:30.475895 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:30.475849 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" event={"ID":"0b87d4af-b1ba-4b14-a7f5-6260136e4151","Type":"ContainerStarted","Data":"d0fd9225b4965d29c3c4d1eba440dc9f69a91f19e187ce5289b61ee1ad61c9ab"} Nov 03 20:59:30.476316 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:30.476081 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 20:59:30.589372 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:30.589309 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" podStartSLOduration=2.750907593 podStartE2EDuration="3.589294956s" totalImagesPullingTime="838.387363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:30.587323201 +0000 UTC m=+719.747395464" watchObservedRunningTime="2026-11-03 20:59:30.589294956 +0000 UTC m=+719.749367262" Nov 03 20:59:32.004418 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:32.004379 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:32.190262 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:32.190223 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:32.820012 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:32.819980 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:36.487261 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:36.487234 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 20:59:37.037510 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.037472 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt"] Nov 03 20:59:37.041810 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.041786 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 20:59:37.094424 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.094394 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c9f7f443-serving-cert\"" Nov 03 20:59:37.096927 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.096887 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt"] Nov 03 20:59:37.138880 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.138845 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb3e4699-5a26-4083-a778-5020fd9200f3-maas-api-tls\") pod \"maas-api-e2e-derive-c9f7f443-698df58d6-84sbt\" (UID: \"fb3e4699-5a26-4083-a778-5020fd9200f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 20:59:37.139095 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.138898 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb3e4699-5a26-4083-a778-5020fd9200f3-openshift-service-ca\") pod \"maas-api-e2e-derive-c9f7f443-698df58d6-84sbt\" (UID: \"fb3e4699-5a26-4083-a778-5020fd9200f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 20:59:37.139095 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.139026 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3e4699-5a26-4083-a778-5020fd9200f3-metrics-certs\") pod \"maas-api-e2e-derive-c9f7f443-698df58d6-84sbt\" (UID: \"fb3e4699-5a26-4083-a778-5020fd9200f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 20:59:37.139095 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.139056 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdfwm\" (UniqueName: \"kubernetes.io/projected/fb3e4699-5a26-4083-a778-5020fd9200f3-kube-api-access-qdfwm\") pod \"maas-api-e2e-derive-c9f7f443-698df58d6-84sbt\" (UID: \"fb3e4699-5a26-4083-a778-5020fd9200f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 20:59:37.240489 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.240445 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3e4699-5a26-4083-a778-5020fd9200f3-metrics-certs\") pod \"maas-api-e2e-derive-c9f7f443-698df58d6-84sbt\" (UID: \"fb3e4699-5a26-4083-a778-5020fd9200f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 20:59:37.240489 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.240492 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdfwm\" (UniqueName: \"kubernetes.io/projected/fb3e4699-5a26-4083-a778-5020fd9200f3-kube-api-access-qdfwm\") pod \"maas-api-e2e-derive-c9f7f443-698df58d6-84sbt\" (UID: \"fb3e4699-5a26-4083-a778-5020fd9200f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 20:59:37.240719 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.240552 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb3e4699-5a26-4083-a778-5020fd9200f3-maas-api-tls\") pod \"maas-api-e2e-derive-c9f7f443-698df58d6-84sbt\" (UID: \"fb3e4699-5a26-4083-a778-5020fd9200f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 20:59:37.240719 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.240594 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb3e4699-5a26-4083-a778-5020fd9200f3-openshift-service-ca\") pod \"maas-api-e2e-derive-c9f7f443-698df58d6-84sbt\" (UID: \"fb3e4699-5a26-4083-a778-5020fd9200f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 20:59:37.240719 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:59:37.240700 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-serving-cert: secret "maas-api-e2e-derive-c9f7f443-serving-cert" not found Nov 03 20:59:37.240828 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:59:37.240779 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3e4699-5a26-4083-a778-5020fd9200f3-maas-api-tls podName:fb3e4699-5a26-4083-a778-5020fd9200f3 nodeName:}" failed. No retries permitted until 2026-11-03 20:59:37.740757912 +0000 UTC m=+726.900830186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb3e4699-5a26-4083-a778-5020fd9200f3-maas-api-tls") pod "maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" (UID: "fb3e4699-5a26-4083-a778-5020fd9200f3") : secret "maas-api-e2e-derive-c9f7f443-serving-cert" not found Nov 03 20:59:37.241304 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.241282 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb3e4699-5a26-4083-a778-5020fd9200f3-openshift-service-ca\") pod \"maas-api-e2e-derive-c9f7f443-698df58d6-84sbt\" (UID: \"fb3e4699-5a26-4083-a778-5020fd9200f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 20:59:37.243021 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.242998 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3e4699-5a26-4083-a778-5020fd9200f3-metrics-certs\") pod \"maas-api-e2e-derive-c9f7f443-698df58d6-84sbt\" (UID: \"fb3e4699-5a26-4083-a778-5020fd9200f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 20:59:37.272927 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.272794 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:37.299259 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.299186 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdfwm\" (UniqueName: \"kubernetes.io/projected/fb3e4699-5a26-4083-a778-5020fd9200f3-kube-api-access-qdfwm\") pod \"maas-api-e2e-derive-c9f7f443-698df58d6-84sbt\" (UID: \"fb3e4699-5a26-4083-a778-5020fd9200f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 20:59:37.457546 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.457493 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:37.641261 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.641126 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:37.745269 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.745230 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb3e4699-5a26-4083-a778-5020fd9200f3-maas-api-tls\") pod \"maas-api-e2e-derive-c9f7f443-698df58d6-84sbt\" (UID: \"fb3e4699-5a26-4083-a778-5020fd9200f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 20:59:37.747814 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.747789 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb3e4699-5a26-4083-a778-5020fd9200f3-maas-api-tls\") pod \"maas-api-e2e-derive-c9f7f443-698df58d6-84sbt\" (UID: \"fb3e4699-5a26-4083-a778-5020fd9200f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 20:59:37.954942 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:37.954879 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 20:59:38.328326 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:38.328240 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt"] Nov 03 20:59:38.334612 ip-10-0-130-134 kubenswrapper[4963]: W1103 20:59:38.334585 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb3e4699_5a26_4083_a778_5020fd9200f3.slice/crio-cc6e1bd54b66e4d7ba728e5d729313b3dc5560d3e45d1270cabb2d10a6da2d68 WatchSource:0}: Error finding container cc6e1bd54b66e4d7ba728e5d729313b3dc5560d3e45d1270cabb2d10a6da2d68: Status 404 returned error can't find the container with id cc6e1bd54b66e4d7ba728e5d729313b3dc5560d3e45d1270cabb2d10a6da2d68 Nov 03 20:59:38.517134 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:38.517099 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" event={"ID":"fb3e4699-5a26-4083-a778-5020fd9200f3","Type":"ContainerStarted","Data":"cc6e1bd54b66e4d7ba728e5d729313b3dc5560d3e45d1270cabb2d10a6da2d68"} Nov 03 20:59:39.523842 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:39.523807 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" event={"ID":"fb3e4699-5a26-4083-a778-5020fd9200f3","Type":"ContainerStarted","Data":"7e218f41a3ff676626915e6c98713c42eee6f86ebaa3883fe7d694f93c5ddbcc"} Nov 03 20:59:39.524266 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:39.523944 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 20:59:39.636351 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:39.636296 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" podStartSLOduration=2.8469763 podStartE2EDuration="3.636280736s" totalImagesPullingTime="789.304436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:39.634985955 +0000 UTC m=+728.795058226" watchObservedRunningTime="2026-11-03 20:59:39.636280736 +0000 UTC m=+728.796353008" Nov 03 20:59:42.007472 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:42.007433 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:42.528269 ip-10-0-130-134 kubenswrapper[4963]: E1103 20:59:42.528227 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 20:59:45.534345 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:45.534314 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 20:59:46.221518 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:46.221466 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:51.150818 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:51.150779 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:56.009474 ip-10-0-130-134 kubenswrapper[4963]: I1103 20:59:56.009432 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:00.190634 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:00.190601 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw"] Nov 03 21:00:00.194395 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:00.194377 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw" Nov 03 21:00:00.244110 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:00.244077 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\"" Nov 03 21:00:00.293539 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:00.293495 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw"] Nov 03 21:00:00.369014 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:00.368972 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j5hq\" (UniqueName: \"kubernetes.io/projected/ece8a8d4-9a9c-4881-a40d-ace86dd37ae9-kube-api-access-4j5hq\") pod \"maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw\" (UID: \"ece8a8d4-9a9c-4881-a40d-ace86dd37ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw" Nov 03 21:00:00.369164 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:00.369051 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece8a8d4-9a9c-4881-a40d-ace86dd37ae9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw\" (UID: \"ece8a8d4-9a9c-4881-a40d-ace86dd37ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw" Nov 03 21:00:00.469791 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:00.469756 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j5hq\" (UniqueName: \"kubernetes.io/projected/ece8a8d4-9a9c-4881-a40d-ace86dd37ae9-kube-api-access-4j5hq\") pod \"maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw\" (UID: \"ece8a8d4-9a9c-4881-a40d-ace86dd37ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw" Nov 03 21:00:00.469999 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:00.469827 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece8a8d4-9a9c-4881-a40d-ace86dd37ae9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw\" (UID: \"ece8a8d4-9a9c-4881-a40d-ace86dd37ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw" Nov 03 21:00:00.470453 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:00.470421 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece8a8d4-9a9c-4881-a40d-ace86dd37ae9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw\" (UID: \"ece8a8d4-9a9c-4881-a40d-ace86dd37ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw" Nov 03 21:00:00.528196 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:00.528165 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j5hq\" (UniqueName: \"kubernetes.io/projected/ece8a8d4-9a9c-4881-a40d-ace86dd37ae9-kube-api-access-4j5hq\") pod \"maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw\" (UID: \"ece8a8d4-9a9c-4881-a40d-ace86dd37ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw" Nov 03 21:00:00.811442 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:00.811357 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw" Nov 03 21:00:00.848803 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:00.848772 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:01.183854 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:01.183775 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw"] Nov 03 21:00:01.190198 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:00:01.190162 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece8a8d4_9a9c_4881_a40d_ace86dd37ae9.slice/crio-24177dd4306362444e0970d0af5e79ffb5bb22ce536af9eaf68da376fb4ccc24 WatchSource:0}: Error finding container 24177dd4306362444e0970d0af5e79ffb5bb22ce536af9eaf68da376fb4ccc24: Status 404 returned error can't find the container with id 24177dd4306362444e0970d0af5e79ffb5bb22ce536af9eaf68da376fb4ccc24 Nov 03 21:00:01.637873 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:01.637830 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw" event={"ID":"ece8a8d4-9a9c-4881-a40d-ace86dd37ae9","Type":"ContainerStarted","Data":"24177dd4306362444e0970d0af5e79ffb5bb22ce536af9eaf68da376fb4ccc24"} Nov 03 21:00:04.382082 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:04.382046 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:04.564831 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:04.564785 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:05.745054 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:05.745018 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:07.002536 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:07.002494 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:07.186178 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:07.186146 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:09.894491 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:09.894454 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:10.850033 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:10.849989 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6"] Nov 03 21:00:10.850258 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:10.850229 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" podUID="0b193aba-8d0d-46f1-b62a-0c2b68c0f102" containerName="maas-api" containerID="cri-o://a2376e46e95d3968c7424b6fafc5c1a04c343bae377cb325d7282356ef92a553" gracePeriod=30 Nov 03 21:00:10.994223 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:10.994179 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:11.126104 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.126074 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 21:00:11.170291 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.170257 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-kube-api-access-mv8q4\" (UniqueName: \"kubernetes.io/projected/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-kube-api-access-mv8q4\") pod \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\" (UID: \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\") " Nov 03 21:00:11.170472 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.170301 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-metrics-certs\") pod \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\" (UID: \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\") " Nov 03 21:00:11.170472 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.170352 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-openshift-service-ca\") pod \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\" (UID: \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\") " Nov 03 21:00:11.170472 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.170449 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-maas-api-tls\") pod \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\" (UID: \"0b193aba-8d0d-46f1-b62a-0c2b68c0f102\") " Nov 03 21:00:11.170655 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.170639 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-openshift-service-ca" pod "0b193aba-8d0d-46f1-b62a-0c2b68c0f102" (UID: "0b193aba-8d0d-46f1-b62a-0c2b68c0f102"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:11.170782 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.170764 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:11.172584 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.172552 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-metrics-certs" pod "0b193aba-8d0d-46f1-b62a-0c2b68c0f102" (UID: "0b193aba-8d0d-46f1-b62a-0c2b68c0f102"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:11.172709 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.172591 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-maas-api-tls" pod "0b193aba-8d0d-46f1-b62a-0c2b68c0f102" (UID: "0b193aba-8d0d-46f1-b62a-0c2b68c0f102"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:11.172709 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.172679 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-kube-api-access-mv8q4" pod "0b193aba-8d0d-46f1-b62a-0c2b68c0f102" (UID: "0b193aba-8d0d-46f1-b62a-0c2b68c0f102"). InnerVolumeSpecName "kube-api-access-mv8q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:11.271329 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.271296 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:11.271329 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.271328 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv8q4\" (UniqueName: \"kubernetes.io/projected/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-kube-api-access-mv8q4\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:11.271609 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.271339 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b193aba-8d0d-46f1-b62a-0c2b68c0f102-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:11.398281 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.398218 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:11.690715 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.690674 4963 generic.go:356] "Generic (PLEG): container finished" podID="0b193aba-8d0d-46f1-b62a-0c2b68c0f102" containerID="a2376e46e95d3968c7424b6fafc5c1a04c343bae377cb325d7282356ef92a553" exitCode=0 Nov 03 21:00:11.690877 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.690735 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" Nov 03 21:00:11.690877 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.690761 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" event={"ID":"0b193aba-8d0d-46f1-b62a-0c2b68c0f102","Type":"ContainerDied","Data":"a2376e46e95d3968c7424b6fafc5c1a04c343bae377cb325d7282356ef92a553"} Nov 03 21:00:11.690877 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.690824 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6" event={"ID":"0b193aba-8d0d-46f1-b62a-0c2b68c0f102","Type":"ContainerDied","Data":"650cbd4178c7a301ff12bcbd0f88841f9080b45f8e2508dff55d6e734f26b153"} Nov 03 21:00:11.690877 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.690853 4963 scope.go:122] "RemoveContainer" containerID="a2376e46e95d3968c7424b6fafc5c1a04c343bae377cb325d7282356ef92a553" Nov 03 21:00:11.692623 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.692599 4963 generic.go:356] "Generic (PLEG): container finished" podID="ece8a8d4-9a9c-4881-a40d-ace86dd37ae9" containerID="9a10f8bada413f9dbfa37f7a3fb5f6b4d3809e45131c55b6c348c272cb8b87ef" exitCode=60 Nov 03 21:00:11.692711 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.692684 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw" event={"ID":"ece8a8d4-9a9c-4881-a40d-ace86dd37ae9","Type":"ContainerDied","Data":"9a10f8bada413f9dbfa37f7a3fb5f6b4d3809e45131c55b6c348c272cb8b87ef"} Nov 03 21:00:11.692949 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.692931 4963 scope.go:122] "RemoveContainer" containerID="9a10f8bada413f9dbfa37f7a3fb5f6b4d3809e45131c55b6c348c272cb8b87ef" Nov 03 21:00:11.706091 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.706069 4963 scope.go:122] "RemoveContainer" containerID="a2376e46e95d3968c7424b6fafc5c1a04c343bae377cb325d7282356ef92a553" Nov 03 21:00:11.706454 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:00:11.706431 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2376e46e95d3968c7424b6fafc5c1a04c343bae377cb325d7282356ef92a553\": container with ID starting with a2376e46e95d3968c7424b6fafc5c1a04c343bae377cb325d7282356ef92a553 not found: ID does not exist" containerID="a2376e46e95d3968c7424b6fafc5c1a04c343bae377cb325d7282356ef92a553" Nov 03 21:00:11.706542 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.706462 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2376e46e95d3968c7424b6fafc5c1a04c343bae377cb325d7282356ef92a553"} err="failed to get container status \"a2376e46e95d3968c7424b6fafc5c1a04c343bae377cb325d7282356ef92a553\": rpc error: code = NotFound desc = could not find container \"a2376e46e95d3968c7424b6fafc5c1a04c343bae377cb325d7282356ef92a553\": container with ID starting with a2376e46e95d3968c7424b6fafc5c1a04c343bae377cb325d7282356ef92a553 not found: ID does not exist" Nov 03 21:00:11.709481 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.709456 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:11.976613 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.976580 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6"] Nov 03 21:00:11.983612 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:11.983574 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a30f61a-5d7f6d655c-qjfh6"] Nov 03 21:00:12.137266 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:12.137233 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:12.518077 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:12.518046 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:12.530054 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:00:12.530013 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:00:12.700728 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:12.700695 4963 generic.go:356] "Generic (PLEG): container finished" podID="ece8a8d4-9a9c-4881-a40d-ace86dd37ae9" containerID="60df9297cd1f0f49ac6726371ef9e7c91145d6133c756a87ab7a42ab4e40f177" exitCode=60 Nov 03 21:00:12.700957 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:12.700765 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw" event={"ID":"ece8a8d4-9a9c-4881-a40d-ace86dd37ae9","Type":"ContainerDied","Data":"60df9297cd1f0f49ac6726371ef9e7c91145d6133c756a87ab7a42ab4e40f177"} Nov 03 21:00:12.700957 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:12.700840 4963 scope.go:122] "RemoveContainer" containerID="9a10f8bada413f9dbfa37f7a3fb5f6b4d3809e45131c55b6c348c272cb8b87ef" Nov 03 21:00:12.701168 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:12.701152 4963 scope.go:122] "RemoveContainer" containerID="60df9297cd1f0f49ac6726371ef9e7c91145d6133c756a87ab7a42ab4e40f177" Nov 03 21:00:12.701407 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:00:12.701385 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw_odh-ai-gateway-infra(ece8a8d4-9a9c-4881-a40d-ace86dd37ae9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw" podUID="ece8a8d4-9a9c-4881-a40d-ace86dd37ae9" Nov 03 21:00:12.728392 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:12.728357 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:12.908467 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:12.908376 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:12.966302 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:12.966270 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b193aba-8d0d-46f1-b62a-0c2b68c0f102" path="/var/lib/kubelet/pods/0b193aba-8d0d-46f1-b62a-0c2b68c0f102/volumes" Nov 03 21:00:13.121484 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:13.121456 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:13.707917 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:13.707882 4963 scope.go:122] "RemoveContainer" containerID="60df9297cd1f0f49ac6726371ef9e7c91145d6133c756a87ab7a42ab4e40f177" Nov 03 21:00:13.708317 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:00:13.708160 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw_odh-ai-gateway-infra(ece8a8d4-9a9c-4881-a40d-ace86dd37ae9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw" podUID="ece8a8d4-9a9c-4881-a40d-ace86dd37ae9" Nov 03 21:00:13.912099 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:13.912059 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:14.092672 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:14.092583 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:14.402714 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:14.402618 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:14.587269 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:14.587233 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:14.777427 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:14.777370 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:14.959010 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:14.958974 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:15.165439 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:15.165343 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:16.696563 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:16.696523 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm"] Nov 03 21:00:16.697038 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:16.696758 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" podUID="0b87d4af-b1ba-4b14-a7f5-6260136e4151" containerName="maas-api" containerID="cri-o://d0fd9225b4965d29c3c4d1eba440dc9f69a91f19e187ce5289b61ee1ad61c9ab" gracePeriod=30 Nov 03 21:00:16.837523 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:16.837480 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:16.971754 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:16.971720 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 21:00:17.021378 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.021333 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b87d4af-b1ba-4b14-a7f5-6260136e4151-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b87d4af-b1ba-4b14-a7f5-6260136e4151-metrics-certs\") pod \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\" (UID: \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\") " Nov 03 21:00:17.021575 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.021388 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b87d4af-b1ba-4b14-a7f5-6260136e4151-kube-api-access-5bwgn\" (UniqueName: \"kubernetes.io/projected/0b87d4af-b1ba-4b14-a7f5-6260136e4151-kube-api-access-5bwgn\") pod \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\" (UID: \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\") " Nov 03 21:00:17.021575 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.021460 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b87d4af-b1ba-4b14-a7f5-6260136e4151-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b87d4af-b1ba-4b14-a7f5-6260136e4151-openshift-service-ca\") pod \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\" (UID: \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\") " Nov 03 21:00:17.021575 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.021488 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b87d4af-b1ba-4b14-a7f5-6260136e4151-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b87d4af-b1ba-4b14-a7f5-6260136e4151-maas-api-tls\") pod \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\" (UID: \"0b87d4af-b1ba-4b14-a7f5-6260136e4151\") " Nov 03 21:00:17.022118 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.021843 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b87d4af-b1ba-4b14-a7f5-6260136e4151-openshift-service-ca" pod "0b87d4af-b1ba-4b14-a7f5-6260136e4151" (UID: "0b87d4af-b1ba-4b14-a7f5-6260136e4151"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:17.023687 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.023659 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b87d4af-b1ba-4b14-a7f5-6260136e4151-kube-api-access-5bwgn" pod "0b87d4af-b1ba-4b14-a7f5-6260136e4151" (UID: "0b87d4af-b1ba-4b14-a7f5-6260136e4151"). InnerVolumeSpecName "kube-api-access-5bwgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:17.023797 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.023720 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b87d4af-b1ba-4b14-a7f5-6260136e4151-metrics-certs" pod "0b87d4af-b1ba-4b14-a7f5-6260136e4151" (UID: "0b87d4af-b1ba-4b14-a7f5-6260136e4151"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:17.023851 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.023813 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b87d4af-b1ba-4b14-a7f5-6260136e4151-maas-api-tls" pod "0b87d4af-b1ba-4b14-a7f5-6260136e4151" (UID: "0b87d4af-b1ba-4b14-a7f5-6260136e4151"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:17.123226 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.123183 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b87d4af-b1ba-4b14-a7f5-6260136e4151-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:17.123399 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.123234 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b87d4af-b1ba-4b14-a7f5-6260136e4151-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:17.123399 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.123250 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b87d4af-b1ba-4b14-a7f5-6260136e4151-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:17.123399 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.123265 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bwgn\" (UniqueName: \"kubernetes.io/projected/0b87d4af-b1ba-4b14-a7f5-6260136e4151-kube-api-access-5bwgn\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:17.358515 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.358424 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:17.538868 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.538835 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:17.731211 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.731173 4963 generic.go:356] "Generic (PLEG): container finished" podID="0b87d4af-b1ba-4b14-a7f5-6260136e4151" containerID="d0fd9225b4965d29c3c4d1eba440dc9f69a91f19e187ce5289b61ee1ad61c9ab" exitCode=0 Nov 03 21:00:17.731704 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.731238 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" Nov 03 21:00:17.731704 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.731257 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" event={"ID":"0b87d4af-b1ba-4b14-a7f5-6260136e4151","Type":"ContainerDied","Data":"d0fd9225b4965d29c3c4d1eba440dc9f69a91f19e187ce5289b61ee1ad61c9ab"} Nov 03 21:00:17.731704 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.731314 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm" event={"ID":"0b87d4af-b1ba-4b14-a7f5-6260136e4151","Type":"ContainerDied","Data":"d440bf12720238395d616f2ac17cdd4986783ac87d60382fb7ce69d5aba04112"} Nov 03 21:00:17.731704 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.731340 4963 scope.go:122] "RemoveContainer" containerID="d0fd9225b4965d29c3c4d1eba440dc9f69a91f19e187ce5289b61ee1ad61c9ab" Nov 03 21:00:17.742991 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.742966 4963 scope.go:122] "RemoveContainer" containerID="d0fd9225b4965d29c3c4d1eba440dc9f69a91f19e187ce5289b61ee1ad61c9ab" Nov 03 21:00:17.743357 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:00:17.743332 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0fd9225b4965d29c3c4d1eba440dc9f69a91f19e187ce5289b61ee1ad61c9ab\": container with ID starting with d0fd9225b4965d29c3c4d1eba440dc9f69a91f19e187ce5289b61ee1ad61c9ab not found: ID does not exist" containerID="d0fd9225b4965d29c3c4d1eba440dc9f69a91f19e187ce5289b61ee1ad61c9ab" Nov 03 21:00:17.743444 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.743366 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fd9225b4965d29c3c4d1eba440dc9f69a91f19e187ce5289b61ee1ad61c9ab"} err="failed to get container status \"d0fd9225b4965d29c3c4d1eba440dc9f69a91f19e187ce5289b61ee1ad61c9ab\": rpc error: code = NotFound desc = could not find container \"d0fd9225b4965d29c3c4d1eba440dc9f69a91f19e187ce5289b61ee1ad61c9ab\": container with ID starting with d0fd9225b4965d29c3c4d1eba440dc9f69a91f19e187ce5289b61ee1ad61c9ab not found: ID does not exist" Nov 03 21:00:17.887985 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.887952 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm"] Nov 03 21:00:17.891803 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:17.891776 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10bfccd7-8467478557-42hcm"] Nov 03 21:00:18.750506 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:18.750472 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:18.939494 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:18.939463 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:18.966118 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:18.966085 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b87d4af-b1ba-4b14-a7f5-6260136e4151" path="/var/lib/kubelet/pods/0b87d4af-b1ba-4b14-a7f5-6260136e4151/volumes" Nov 03 21:00:19.131209 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:19.131123 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:19.371554 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:19.371519 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:19.567130 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:19.567098 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:19.751354 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:19.751322 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:20.808348 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:20.808312 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:22.320898 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:22.320827 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:22.642163 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:22.642071 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:22.967385 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:22.967350 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:23.152641 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:23.152606 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:23.658697 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:23.658653 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:23.846557 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:23.846524 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:24.391078 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:24.391048 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:24.960966 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:24.960900 4963 scope.go:122] "RemoveContainer" containerID="60df9297cd1f0f49ac6726371ef9e7c91145d6133c756a87ab7a42ab4e40f177" Nov 03 21:00:25.180860 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:25.180826 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:25.361870 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:25.361836 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:25.548010 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:25.547876 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:25.733126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:25.733076 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:25.777132 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:25.777086 4963 generic.go:356] "Generic (PLEG): container finished" podID="ece8a8d4-9a9c-4881-a40d-ace86dd37ae9" containerID="1e247ae1dbf9dce41fc95ce7028607a258178251ece62eab675f1d594d8cc3bc" exitCode=60 Nov 03 21:00:25.777308 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:25.777164 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw" event={"ID":"ece8a8d4-9a9c-4881-a40d-ace86dd37ae9","Type":"ContainerDied","Data":"1e247ae1dbf9dce41fc95ce7028607a258178251ece62eab675f1d594d8cc3bc"} Nov 03 21:00:25.777308 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:25.777239 4963 scope.go:122] "RemoveContainer" containerID="60df9297cd1f0f49ac6726371ef9e7c91145d6133c756a87ab7a42ab4e40f177" Nov 03 21:00:25.779038 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:25.779016 4963 scope.go:122] "RemoveContainer" containerID="1e247ae1dbf9dce41fc95ce7028607a258178251ece62eab675f1d594d8cc3bc" Nov 03 21:00:25.779327 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:00:25.779293 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw_odh-ai-gateway-infra(ece8a8d4-9a9c-4881-a40d-ace86dd37ae9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw" podUID="ece8a8d4-9a9c-4881-a40d-ace86dd37ae9" Nov 03 21:00:25.917804 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:25.917723 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:26.107468 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:26.107438 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:26.896275 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:26.896240 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw"] Nov 03 21:00:27.056388 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:27.056355 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw" Nov 03 21:00:27.074554 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:27.074520 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:27.115221 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:27.115185 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ece8a8d4-9a9c-4881-a40d-ace86dd37ae9-kube-api-access-4j5hq\" (UniqueName: \"kubernetes.io/projected/ece8a8d4-9a9c-4881-a40d-ace86dd37ae9-kube-api-access-4j5hq\") pod \"ece8a8d4-9a9c-4881-a40d-ace86dd37ae9\" (UID: \"ece8a8d4-9a9c-4881-a40d-ace86dd37ae9\") " Nov 03 21:00:27.115712 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:27.115257 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ece8a8d4-9a9c-4881-a40d-ace86dd37ae9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece8a8d4-9a9c-4881-a40d-ace86dd37ae9-maas-api-service-ca\") pod \"ece8a8d4-9a9c-4881-a40d-ace86dd37ae9\" (UID: \"ece8a8d4-9a9c-4881-a40d-ace86dd37ae9\") " Nov 03 21:00:27.115712 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:27.115622 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece8a8d4-9a9c-4881-a40d-ace86dd37ae9-maas-api-service-ca" pod "ece8a8d4-9a9c-4881-a40d-ace86dd37ae9" (UID: "ece8a8d4-9a9c-4881-a40d-ace86dd37ae9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:27.117524 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:27.117501 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece8a8d4-9a9c-4881-a40d-ace86dd37ae9-kube-api-access-4j5hq" pod "ece8a8d4-9a9c-4881-a40d-ace86dd37ae9" (UID: "ece8a8d4-9a9c-4881-a40d-ace86dd37ae9"). InnerVolumeSpecName "kube-api-access-4j5hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:27.216482 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:27.216450 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j5hq\" (UniqueName: \"kubernetes.io/projected/ece8a8d4-9a9c-4881-a40d-ace86dd37ae9-kube-api-access-4j5hq\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:27.216482 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:27.216481 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece8a8d4-9a9c-4881-a40d-ace86dd37ae9-maas-api-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:27.316735 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:27.316649 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:27.796700 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:27.796670 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw" Nov 03 21:00:27.796895 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:27.796664 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw" event={"ID":"ece8a8d4-9a9c-4881-a40d-ace86dd37ae9","Type":"ContainerDied","Data":"24177dd4306362444e0970d0af5e79ffb5bb22ce536af9eaf68da376fb4ccc24"} Nov 03 21:00:27.796895 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:27.796854 4963 scope.go:122] "RemoveContainer" containerID="1e247ae1dbf9dce41fc95ce7028607a258178251ece62eab675f1d594d8cc3bc" Nov 03 21:00:27.854280 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:27.854240 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:27.952376 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:27.952346 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw"] Nov 03 21:00:27.999691 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:27.999656 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-c9f7f443-29895660-d9smw"] Nov 03 21:00:28.966330 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:28.966298 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ece8a8d4-9a9c-4881-a40d-ace86dd37ae9" path="/var/lib/kubelet/pods/ece8a8d4-9a9c-4881-a40d-ace86dd37ae9/volumes" Nov 03 21:00:29.343773 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:29.343670 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:29.577779 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:29.577746 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:29.761044 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:29.761014 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:29.945978 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:29.945941 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:30.549549 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:30.549512 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:31.079802 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:31.079752 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:31.264483 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:31.264433 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:31.445998 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:31.445961 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:31.957223 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:31.957183 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:32.326858 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:32.326772 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:33.150658 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:33.150622 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:34.725771 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:34.725731 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:34.916021 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:34.915988 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:35.098455 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:35.098372 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:35.465946 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:35.465889 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:35.649995 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:35.649960 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:38.094001 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:38.093896 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:38.302427 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:38.302388 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:39.804494 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:39.804460 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:40.536982 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:40.536921 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:40.993447 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:40.993350 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:42.528116 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:00:42.528073 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:00:43.747058 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:43.747022 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt"] Nov 03 21:00:43.747484 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:43.747253 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" podUID="fb3e4699-5a26-4083-a778-5020fd9200f3" containerName="maas-api" containerID="cri-o://7e218f41a3ff676626915e6c98713c42eee6f86ebaa3883fe7d694f93c5ddbcc" gracePeriod=30 Nov 03 21:00:43.880692 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:43.880658 4963 generic.go:356] "Generic (PLEG): container finished" podID="fb3e4699-5a26-4083-a778-5020fd9200f3" containerID="7e218f41a3ff676626915e6c98713c42eee6f86ebaa3883fe7d694f93c5ddbcc" exitCode=0 Nov 03 21:00:43.880857 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:43.880727 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" event={"ID":"fb3e4699-5a26-4083-a778-5020fd9200f3","Type":"ContainerDied","Data":"7e218f41a3ff676626915e6c98713c42eee6f86ebaa3883fe7d694f93c5ddbcc"} Nov 03 21:00:43.896133 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:43.896103 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:44.018258 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:44.018229 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 21:00:44.187278 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:44.187247 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb3e4699-5a26-4083-a778-5020fd9200f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb3e4699-5a26-4083-a778-5020fd9200f3-openshift-service-ca\") pod \"fb3e4699-5a26-4083-a778-5020fd9200f3\" (UID: \"fb3e4699-5a26-4083-a778-5020fd9200f3\") " Nov 03 21:00:44.187278 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:44.187280 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb3e4699-5a26-4083-a778-5020fd9200f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb3e4699-5a26-4083-a778-5020fd9200f3-maas-api-tls\") pod \"fb3e4699-5a26-4083-a778-5020fd9200f3\" (UID: \"fb3e4699-5a26-4083-a778-5020fd9200f3\") " Nov 03 21:00:44.187539 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:44.187320 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb3e4699-5a26-4083-a778-5020fd9200f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3e4699-5a26-4083-a778-5020fd9200f3-metrics-certs\") pod \"fb3e4699-5a26-4083-a778-5020fd9200f3\" (UID: \"fb3e4699-5a26-4083-a778-5020fd9200f3\") " Nov 03 21:00:44.187539 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:44.187368 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb3e4699-5a26-4083-a778-5020fd9200f3-kube-api-access-qdfwm\" (UniqueName: \"kubernetes.io/projected/fb3e4699-5a26-4083-a778-5020fd9200f3-kube-api-access-qdfwm\") pod \"fb3e4699-5a26-4083-a778-5020fd9200f3\" (UID: \"fb3e4699-5a26-4083-a778-5020fd9200f3\") " Nov 03 21:00:44.187719 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:44.187668 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb3e4699-5a26-4083-a778-5020fd9200f3-openshift-service-ca" pod "fb3e4699-5a26-4083-a778-5020fd9200f3" (UID: "fb3e4699-5a26-4083-a778-5020fd9200f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:44.189461 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:44.189428 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb3e4699-5a26-4083-a778-5020fd9200f3-maas-api-tls" pod "fb3e4699-5a26-4083-a778-5020fd9200f3" (UID: "fb3e4699-5a26-4083-a778-5020fd9200f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:44.189461 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:44.189456 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb3e4699-5a26-4083-a778-5020fd9200f3-metrics-certs" pod "fb3e4699-5a26-4083-a778-5020fd9200f3" (UID: "fb3e4699-5a26-4083-a778-5020fd9200f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:44.189611 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:44.189534 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3e4699-5a26-4083-a778-5020fd9200f3-kube-api-access-qdfwm" pod "fb3e4699-5a26-4083-a778-5020fd9200f3" (UID: "fb3e4699-5a26-4083-a778-5020fd9200f3"). InnerVolumeSpecName "kube-api-access-qdfwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:44.288065 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:44.287975 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb3e4699-5a26-4083-a778-5020fd9200f3-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:44.288065 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:44.288010 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdfwm\" (UniqueName: \"kubernetes.io/projected/fb3e4699-5a26-4083-a778-5020fd9200f3-kube-api-access-qdfwm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:44.288065 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:44.288021 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb3e4699-5a26-4083-a778-5020fd9200f3-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:44.288065 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:44.288030 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb3e4699-5a26-4083-a778-5020fd9200f3-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:44.887300 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:44.887266 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" Nov 03 21:00:44.887734 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:44.887293 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt" event={"ID":"fb3e4699-5a26-4083-a778-5020fd9200f3","Type":"ContainerDied","Data":"cc6e1bd54b66e4d7ba728e5d729313b3dc5560d3e45d1270cabb2d10a6da2d68"} Nov 03 21:00:44.887734 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:44.887367 4963 scope.go:122] "RemoveContainer" containerID="7e218f41a3ff676626915e6c98713c42eee6f86ebaa3883fe7d694f93c5ddbcc" Nov 03 21:00:45.043537 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.043502 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt"] Nov 03 21:00:45.091121 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.091089 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c9f7f443-698df58d6-84sbt"] Nov 03 21:00:45.237936 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.237886 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph"] Nov 03 21:00:45.238473 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.238451 4963 cpu_manager.go:414] "containerMap: removing container" podUID="fb3e4699-5a26-4083-a778-5020fd9200f3" containerName="maas-api" Nov 03 21:00:45.238473 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.238472 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb3e4699-5a26-4083-a778-5020fd9200f3" containerName="maas-api" Nov 03 21:00:45.238648 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.238488 4963 cpu_manager.go:414] "containerMap: removing container" podUID="ece8a8d4-9a9c-4881-a40d-ace86dd37ae9" containerName="cleanup" Nov 03 21:00:45.238648 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.238495 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ece8a8d4-9a9c-4881-a40d-ace86dd37ae9" containerName="cleanup" Nov 03 21:00:45.238648 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.238514 4963 cpu_manager.go:414] "containerMap: removing container" podUID="ece8a8d4-9a9c-4881-a40d-ace86dd37ae9" containerName="cleanup" Nov 03 21:00:45.238648 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.238523 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ece8a8d4-9a9c-4881-a40d-ace86dd37ae9" containerName="cleanup" Nov 03 21:00:45.238648 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.238548 4963 cpu_manager.go:414] "containerMap: removing container" podUID="ece8a8d4-9a9c-4881-a40d-ace86dd37ae9" containerName="cleanup" Nov 03 21:00:45.238648 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.238556 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ece8a8d4-9a9c-4881-a40d-ace86dd37ae9" containerName="cleanup" Nov 03 21:00:45.238648 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.238582 4963 cpu_manager.go:414] "containerMap: removing container" podUID="0b193aba-8d0d-46f1-b62a-0c2b68c0f102" containerName="maas-api" Nov 03 21:00:45.238648 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.238589 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b193aba-8d0d-46f1-b62a-0c2b68c0f102" containerName="maas-api" Nov 03 21:00:45.238648 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.238600 4963 cpu_manager.go:414] "containerMap: removing container" podUID="0b87d4af-b1ba-4b14-a7f5-6260136e4151" containerName="maas-api" Nov 03 21:00:45.238648 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.238608 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b87d4af-b1ba-4b14-a7f5-6260136e4151" containerName="maas-api" Nov 03 21:00:45.239189 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.238720 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b193aba-8d0d-46f1-b62a-0c2b68c0f102" containerName="maas-api" Nov 03 21:00:45.239189 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.238732 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b87d4af-b1ba-4b14-a7f5-6260136e4151" containerName="maas-api" Nov 03 21:00:45.239189 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.238743 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb3e4699-5a26-4083-a778-5020fd9200f3" containerName="maas-api" Nov 03 21:00:45.239189 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.238754 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="ece8a8d4-9a9c-4881-a40d-ace86dd37ae9" containerName="cleanup" Nov 03 21:00:45.239189 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.238764 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="ece8a8d4-9a9c-4881-a40d-ace86dd37ae9" containerName="cleanup" Nov 03 21:00:45.239189 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.238775 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="ece8a8d4-9a9c-4881-a40d-ace86dd37ae9" containerName="cleanup" Nov 03 21:00:45.243549 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.243528 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 21:00:45.275515 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.275482 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:45.297147 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.297093 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-e84357-serving-cert\"" Nov 03 21:00:45.297147 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.297143 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:00:45.297383 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.297206 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:00:45.297383 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.297271 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6whcf\"" Nov 03 21:00:45.297383 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.297331 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 21:00:45.300196 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.300168 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph"] Nov 03 21:00:45.398513 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.398470 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/679af1f2-dd8a-4af5-a8d8-518a4a0db304-metrics-certs\") pod \"maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph\" (UID: \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 21:00:45.398513 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.398508 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/679af1f2-dd8a-4af5-a8d8-518a4a0db304-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph\" (UID: \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 21:00:45.398763 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.398671 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86b9b\" (UniqueName: \"kubernetes.io/projected/679af1f2-dd8a-4af5-a8d8-518a4a0db304-kube-api-access-86b9b\") pod \"maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph\" (UID: \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 21:00:45.398763 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.398727 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679af1f2-dd8a-4af5-a8d8-518a4a0db304-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph\" (UID: \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 21:00:45.459542 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.459512 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:45.499594 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.499507 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/679af1f2-dd8a-4af5-a8d8-518a4a0db304-metrics-certs\") pod \"maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph\" (UID: \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 21:00:45.499594 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.499548 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/679af1f2-dd8a-4af5-a8d8-518a4a0db304-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph\" (UID: \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 21:00:45.499815 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.499625 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86b9b\" (UniqueName: \"kubernetes.io/projected/679af1f2-dd8a-4af5-a8d8-518a4a0db304-kube-api-access-86b9b\") pod \"maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph\" (UID: \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 21:00:45.499815 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.499668 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679af1f2-dd8a-4af5-a8d8-518a4a0db304-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph\" (UID: \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 21:00:45.499815 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:00:45.499767 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-serving-cert: secret "maas-api-e2e-worker-w4-e84357-serving-cert" not found Nov 03 21:00:45.500001 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:00:45.499840 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/679af1f2-dd8a-4af5-a8d8-518a4a0db304-maas-api-tls podName:679af1f2-dd8a-4af5-a8d8-518a4a0db304 nodeName:}" failed. No retries permitted until 2026-11-03 21:00:45.999818539 +0000 UTC m=+795.159890809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/679af1f2-dd8a-4af5-a8d8-518a4a0db304-maas-api-tls") pod "maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" (UID: "679af1f2-dd8a-4af5-a8d8-518a4a0db304") : secret "maas-api-e2e-worker-w4-e84357-serving-cert" not found Nov 03 21:00:45.500389 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.500364 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/679af1f2-dd8a-4af5-a8d8-518a4a0db304-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph\" (UID: \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 21:00:45.502206 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.502188 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/679af1f2-dd8a-4af5-a8d8-518a4a0db304-metrics-certs\") pod \"maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph\" (UID: \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 21:00:45.558883 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.558845 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86b9b\" (UniqueName: \"kubernetes.io/projected/679af1f2-dd8a-4af5-a8d8-518a4a0db304-kube-api-access-86b9b\") pod \"maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph\" (UID: \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 21:00:45.640626 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.640589 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:45.829549 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:45.829471 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:46.005128 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:46.005087 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679af1f2-dd8a-4af5-a8d8-518a4a0db304-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph\" (UID: \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 21:00:46.007701 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:46.007669 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679af1f2-dd8a-4af5-a8d8-518a4a0db304-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph\" (UID: \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 21:00:46.156383 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:46.156287 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 21:00:46.519355 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:46.519325 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:46.529016 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:46.528987 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph"] Nov 03 21:00:46.535525 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:00:46.535491 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679af1f2_dd8a_4af5_a8d8_518a4a0db304.slice/crio-dbc5fd2d189f183270cd4a1d5f99bbaaf000018b0c950b1ffc5f0d7509051a83 WatchSource:0}: Error finding container dbc5fd2d189f183270cd4a1d5f99bbaaf000018b0c950b1ffc5f0d7509051a83: Status 404 returned error can't find the container with id dbc5fd2d189f183270cd4a1d5f99bbaaf000018b0c950b1ffc5f0d7509051a83 Nov 03 21:00:46.785652 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:46.785551 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:46.900059 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:46.900022 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" event={"ID":"679af1f2-dd8a-4af5-a8d8-518a4a0db304","Type":"ContainerStarted","Data":"dbc5fd2d189f183270cd4a1d5f99bbaaf000018b0c950b1ffc5f0d7509051a83"} Nov 03 21:00:46.973204 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:46.973167 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb3e4699-5a26-4083-a778-5020fd9200f3" path="/var/lib/kubelet/pods/fb3e4699-5a26-4083-a778-5020fd9200f3/volumes" Nov 03 21:00:47.116565 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:47.116467 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:47.906529 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:47.906435 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" event={"ID":"679af1f2-dd8a-4af5-a8d8-518a4a0db304","Type":"ContainerStarted","Data":"f9b9517eaf0e448bc6c7053160386381e58cfa4a585aefce648f50ead8286256"} Nov 03 21:00:47.906529 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:47.906495 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 21:00:47.966688 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:47.966657 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:48.021128 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:48.021072 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" podStartSLOduration=2.1115628969999998 podStartE2EDuration="3.021057237s" totalImagesPullingTime="909.49434ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:48.019157926 +0000 UTC m=+797.179230200" watchObservedRunningTime="2026-11-03 21:00:48.021057237 +0000 UTC m=+797.181129500" Nov 03 21:00:48.634571 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:48.634538 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:48.824255 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:48.824222 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:50.502399 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:50.502362 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:53.316499 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:53.316457 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:53.507439 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:53.507405 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:53.917857 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:53.917830 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 21:00:55.312845 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:55.312811 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:55.493029 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:55.492997 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:59.824998 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:00:59.824968 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:00.013575 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:00.013546 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:00.198626 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:00.198591 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:00.392529 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:00.392493 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:00.574606 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:00.574526 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:04.249886 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.249850 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br"] Nov 03 21:01:04.254279 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.254251 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.354344 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.354301 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-320f60\"/\"openshift-service-ca.crt\"" Nov 03 21:01:04.354344 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.354304 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-320f60\"/\"simulator-w3-320f60-kserve-self-signed-certs\"" Nov 03 21:01:04.361611 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.354799 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-320f60\"/\"kube-root-ca.crt\"" Nov 03 21:01:04.361611 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.354820 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-320f60\"/\"default-dockercfg-mhpg2\"" Nov 03 21:01:04.361611 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.358837 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br"] Nov 03 21:01:04.368849 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.368822 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-home\") pod \"simulator-w3-320f60-kserve-6858fcc8bf-4b2br\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.369049 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.368854 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f5464f-5d94-499e-9fee-7553aa1e78a4-tls-certs\") pod \"simulator-w3-320f60-kserve-6858fcc8bf-4b2br\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.369049 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.368880 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-model-cache\") pod \"simulator-w3-320f60-kserve-6858fcc8bf-4b2br\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.369049 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.368967 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-dshm\") pod \"simulator-w3-320f60-kserve-6858fcc8bf-4b2br\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.369213 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.369123 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sddt2\" (UniqueName: \"kubernetes.io/projected/a3f5464f-5d94-499e-9fee-7553aa1e78a4-kube-api-access-sddt2\") pod \"simulator-w3-320f60-kserve-6858fcc8bf-4b2br\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.369213 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.369191 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-tmp-dir\") pod \"simulator-w3-320f60-kserve-6858fcc8bf-4b2br\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.376997 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.376970 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:04.470690 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.470654 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sddt2\" (UniqueName: \"kubernetes.io/projected/a3f5464f-5d94-499e-9fee-7553aa1e78a4-kube-api-access-sddt2\") pod \"simulator-w3-320f60-kserve-6858fcc8bf-4b2br\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.470898 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.470712 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-tmp-dir\") pod \"simulator-w3-320f60-kserve-6858fcc8bf-4b2br\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.470898 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.470778 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-home\") pod \"simulator-w3-320f60-kserve-6858fcc8bf-4b2br\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.470898 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.470798 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f5464f-5d94-499e-9fee-7553aa1e78a4-tls-certs\") pod \"simulator-w3-320f60-kserve-6858fcc8bf-4b2br\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.470898 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.470826 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-model-cache\") pod \"simulator-w3-320f60-kserve-6858fcc8bf-4b2br\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.470898 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.470849 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-dshm\") pod \"simulator-w3-320f60-kserve-6858fcc8bf-4b2br\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.471316 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.471269 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-home\") pod \"simulator-w3-320f60-kserve-6858fcc8bf-4b2br\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.471400 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.471322 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-tmp-dir\") pod \"simulator-w3-320f60-kserve-6858fcc8bf-4b2br\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.471400 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.471352 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-model-cache\") pod \"simulator-w3-320f60-kserve-6858fcc8bf-4b2br\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.473294 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.473260 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-dshm\") pod \"simulator-w3-320f60-kserve-6858fcc8bf-4b2br\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.473562 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.473543 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f5464f-5d94-499e-9fee-7553aa1e78a4-tls-certs\") pod \"simulator-w3-320f60-kserve-6858fcc8bf-4b2br\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.529350 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.529265 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sddt2\" (UniqueName: \"kubernetes.io/projected/a3f5464f-5d94-499e-9fee-7553aa1e78a4-kube-api-access-sddt2\") pod \"simulator-w3-320f60-kserve-6858fcc8bf-4b2br\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.566172 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.566124 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:04.944948 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.944914 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br"] Nov 03 21:01:04.951010 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:01:04.950975 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3f5464f_5d94_499e_9fee_7553aa1e78a4.slice/crio-3af4120146630ca8cff9b1ec76f1d052a1b88e7327cee0f27942e6948d8e79d0 WatchSource:0}: Error finding container 3af4120146630ca8cff9b1ec76f1d052a1b88e7327cee0f27942e6948d8e79d0: Status 404 returned error can't find the container with id 3af4120146630ca8cff9b1ec76f1d052a1b88e7327cee0f27942e6948d8e79d0 Nov 03 21:01:04.996363 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:04.996329 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" event={"ID":"a3f5464f-5d94-499e-9fee-7553aa1e78a4","Type":"ContainerStarted","Data":"3af4120146630ca8cff9b1ec76f1d052a1b88e7327cee0f27942e6948d8e79d0"} Nov 03 21:01:05.144958 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:05.144896 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:05.335763 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:05.335673 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:05.520823 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:05.520791 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:08.014045 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:08.014008 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" event={"ID":"a3f5464f-5d94-499e-9fee-7553aa1e78a4","Type":"ContainerStarted","Data":"8656024f417dfca40269a9d4d37d6e355bf39e0745d28858891b91fcb5e0821a"} Nov 03 21:01:08.127721 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:08.127664 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" podStartSLOduration=2.058739564 podStartE2EDuration="4.127649889s" totalImagesPullingTime="2.068910325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:08.126824695 +0000 UTC m=+817.286896969" watchObservedRunningTime="2026-11-03 21:01:08.127649889 +0000 UTC m=+817.287722160" Nov 03 21:01:08.566614 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:08.566562 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:10.005476 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:10.005439 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:11.360385 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:11.360347 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:11.542627 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:11.542596 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:12.527845 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:01:12.527809 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:01:14.508155 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:14.508122 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:14.566629 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:14.566590 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:14.579578 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:14.579546 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:15.061154 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:15.061125 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:01:16.055629 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:16.055597 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:16.292106 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:16.292075 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:19.034294 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.034258 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b"] Nov 03 21:01:19.038035 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.038003 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 21:01:19.114079 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.114042 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwpzv\" (UniqueName: \"kubernetes.io/projected/62344e41-1f57-4ca4-9805-668cec58dd3b-kube-api-access-lwpzv\") pod \"maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b\" (UID: \"62344e41-1f57-4ca4-9805-668cec58dd3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 21:01:19.114250 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.114115 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62344e41-1f57-4ca4-9805-668cec58dd3b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b\" (UID: \"62344e41-1f57-4ca4-9805-668cec58dd3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 21:01:19.114250 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.114184 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62344e41-1f57-4ca4-9805-668cec58dd3b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b\" (UID: \"62344e41-1f57-4ca4-9805-668cec58dd3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 21:01:19.114250 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.114212 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62344e41-1f57-4ca4-9805-668cec58dd3b-metrics-certs\") pod \"maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b\" (UID: \"62344e41-1f57-4ca4-9805-668cec58dd3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 21:01:19.134959 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.134898 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-a41ad0-serving-cert\"" Nov 03 21:01:19.137073 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.137050 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b"] Nov 03 21:01:19.215270 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.215223 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62344e41-1f57-4ca4-9805-668cec58dd3b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b\" (UID: \"62344e41-1f57-4ca4-9805-668cec58dd3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 21:01:19.215463 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.215294 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62344e41-1f57-4ca4-9805-668cec58dd3b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b\" (UID: \"62344e41-1f57-4ca4-9805-668cec58dd3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 21:01:19.215463 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.215325 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62344e41-1f57-4ca4-9805-668cec58dd3b-metrics-certs\") pod \"maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b\" (UID: \"62344e41-1f57-4ca4-9805-668cec58dd3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 21:01:19.215463 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.215393 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwpzv\" (UniqueName: \"kubernetes.io/projected/62344e41-1f57-4ca4-9805-668cec58dd3b-kube-api-access-lwpzv\") pod \"maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b\" (UID: \"62344e41-1f57-4ca4-9805-668cec58dd3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 21:01:19.215463 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:01:19.215397 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-serving-cert: secret "maas-api-e2e-worker-w1-a41ad0-serving-cert" not found Nov 03 21:01:19.215628 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:01:19.215486 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62344e41-1f57-4ca4-9805-668cec58dd3b-maas-api-tls podName:62344e41-1f57-4ca4-9805-668cec58dd3b nodeName:}" failed. No retries permitted until 2026-11-03 21:01:19.715463912 +0000 UTC m=+828.875536174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/62344e41-1f57-4ca4-9805-668cec58dd3b-maas-api-tls") pod "maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" (UID: "62344e41-1f57-4ca4-9805-668cec58dd3b") : secret "maas-api-e2e-worker-w1-a41ad0-serving-cert" not found Nov 03 21:01:19.215992 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.215974 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62344e41-1f57-4ca4-9805-668cec58dd3b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b\" (UID: \"62344e41-1f57-4ca4-9805-668cec58dd3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 21:01:19.218020 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.217999 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62344e41-1f57-4ca4-9805-668cec58dd3b-metrics-certs\") pod \"maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b\" (UID: \"62344e41-1f57-4ca4-9805-668cec58dd3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 21:01:19.260988 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.260955 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:19.275183 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.275153 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwpzv\" (UniqueName: \"kubernetes.io/projected/62344e41-1f57-4ca4-9805-668cec58dd3b-kube-api-access-lwpzv\") pod \"maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b\" (UID: \"62344e41-1f57-4ca4-9805-668cec58dd3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 21:01:19.445133 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.445098 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:19.628358 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.628315 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:19.719949 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.719831 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62344e41-1f57-4ca4-9805-668cec58dd3b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b\" (UID: \"62344e41-1f57-4ca4-9805-668cec58dd3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 21:01:19.722416 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.722393 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62344e41-1f57-4ca4-9805-668cec58dd3b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b\" (UID: \"62344e41-1f57-4ca4-9805-668cec58dd3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 21:01:19.950642 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:19.950602 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 21:01:20.321515 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:20.321427 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b"] Nov 03 21:01:20.327492 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:01:20.327465 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62344e41_1f57_4ca4_9805_668cec58dd3b.slice/crio-fb4779da15349452dfd1e4910cdf0f37564db733cdbfe1729147ca05fdcbac9d WatchSource:0}: Error finding container fb4779da15349452dfd1e4910cdf0f37564db733cdbfe1729147ca05fdcbac9d: Status 404 returned error can't find the container with id fb4779da15349452dfd1e4910cdf0f37564db733cdbfe1729147ca05fdcbac9d Nov 03 21:01:21.083077 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:21.083039 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" event={"ID":"62344e41-1f57-4ca4-9805-668cec58dd3b","Type":"ContainerStarted","Data":"fb4779da15349452dfd1e4910cdf0f37564db733cdbfe1729147ca05fdcbac9d"} Nov 03 21:01:22.097299 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:22.097252 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" event={"ID":"62344e41-1f57-4ca4-9805-668cec58dd3b","Type":"ContainerStarted","Data":"e69b59c4261aae23748dc570538e05053e2bc9cde902dba552e80f9dead8beaa"} Nov 03 21:01:22.097768 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:22.097556 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 21:01:22.212950 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:22.212874 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" podStartSLOduration=3.294604243 podStartE2EDuration="4.212858695s" totalImagesPullingTime="918.254452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:22.21057914 +0000 UTC m=+831.370651412" watchObservedRunningTime="2026-11-03 21:01:22.212858695 +0000 UTC m=+831.372930967" Nov 03 21:01:23.996161 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:23.996121 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:25.667020 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:25.666989 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:27.198415 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:27.198376 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:28.107947 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:28.107900 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 21:01:28.605427 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:28.605392 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:31.817024 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:31.816982 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t"] Nov 03 21:01:31.821770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:31.821747 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:31.915663 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:31.915625 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t"] Nov 03 21:01:31.920724 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:31.920705 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e84357\"/\"kube-root-ca.crt\"" Nov 03 21:01:31.920864 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:31.920842 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e84357\"/\"default-dockercfg-q4tcx\"" Nov 03 21:01:31.920943 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:31.920842 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e84357\"/\"simulator-w4-e84357-kserve-self-signed-certs\"" Nov 03 21:01:31.920997 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:31.920936 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e84357\"/\"openshift-service-ca.crt\"" Nov 03 21:01:31.946297 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:31.946266 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-tmp-dir\") pod \"simulator-w4-e84357-kserve-5fd8c78999-zgv4t\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:31.946379 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:31.946311 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1edcbec2-0930-4b49-b14a-2d21568b5318-tls-certs\") pod \"simulator-w4-e84357-kserve-5fd8c78999-zgv4t\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:31.946442 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:31.946421 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-home\") pod \"simulator-w4-e84357-kserve-5fd8c78999-zgv4t\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:31.946546 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:31.946532 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-dshm\") pod \"simulator-w4-e84357-kserve-5fd8c78999-zgv4t\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:31.946600 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:31.946562 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-model-cache\") pod \"simulator-w4-e84357-kserve-5fd8c78999-zgv4t\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:31.946600 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:31.946581 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsp25\" (UniqueName: \"kubernetes.io/projected/1edcbec2-0930-4b49-b14a-2d21568b5318-kube-api-access-rsp25\") pod \"simulator-w4-e84357-kserve-5fd8c78999-zgv4t\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:31.951147 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:31.951107 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:32.047410 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:32.047371 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-model-cache\") pod \"simulator-w4-e84357-kserve-5fd8c78999-zgv4t\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:32.047575 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:32.047415 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsp25\" (UniqueName: \"kubernetes.io/projected/1edcbec2-0930-4b49-b14a-2d21568b5318-kube-api-access-rsp25\") pod \"simulator-w4-e84357-kserve-5fd8c78999-zgv4t\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:32.047575 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:32.047525 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-tmp-dir\") pod \"simulator-w4-e84357-kserve-5fd8c78999-zgv4t\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:32.047575 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:32.047571 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1edcbec2-0930-4b49-b14a-2d21568b5318-tls-certs\") pod \"simulator-w4-e84357-kserve-5fd8c78999-zgv4t\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:32.047765 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:32.047636 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-home\") pod \"simulator-w4-e84357-kserve-5fd8c78999-zgv4t\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:32.047765 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:32.047725 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-dshm\") pod \"simulator-w4-e84357-kserve-5fd8c78999-zgv4t\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:32.047869 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:32.047833 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-model-cache\") pod \"simulator-w4-e84357-kserve-5fd8c78999-zgv4t\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:32.048048 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:32.048023 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-tmp-dir\") pod \"simulator-w4-e84357-kserve-5fd8c78999-zgv4t\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:32.048207 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:32.048181 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-home\") pod \"simulator-w4-e84357-kserve-5fd8c78999-zgv4t\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:32.050049 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:32.050022 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-dshm\") pod \"simulator-w4-e84357-kserve-5fd8c78999-zgv4t\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:32.050497 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:32.050473 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1edcbec2-0930-4b49-b14a-2d21568b5318-tls-certs\") pod \"simulator-w4-e84357-kserve-5fd8c78999-zgv4t\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:32.105836 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:32.105760 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsp25\" (UniqueName: \"kubernetes.io/projected/1edcbec2-0930-4b49-b14a-2d21568b5318-kube-api-access-rsp25\") pod \"simulator-w4-e84357-kserve-5fd8c78999-zgv4t\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:32.133498 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:32.133462 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:32.513171 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:32.513134 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t"] Nov 03 21:01:32.517551 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:01:32.517522 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1edcbec2_0930_4b49_b14a_2d21568b5318.slice/crio-0658bf15bd1dd2ab8141920f3f78c9f190e3063a8a54c9dbec9ef0e850dce0ca WatchSource:0}: Error finding container 0658bf15bd1dd2ab8141920f3f78c9f190e3063a8a54c9dbec9ef0e850dce0ca: Status 404 returned error can't find the container with id 0658bf15bd1dd2ab8141920f3f78c9f190e3063a8a54c9dbec9ef0e850dce0ca Nov 03 21:01:33.115035 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:33.115001 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:33.156478 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:33.156440 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" event={"ID":"1edcbec2-0930-4b49-b14a-2d21568b5318","Type":"ContainerStarted","Data":"41d64440d3501f8d1efb054654d3c8390c1694cece8a0ea5050aef97861367c5"} Nov 03 21:01:33.156478 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:33.156492 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" event={"ID":"1edcbec2-0930-4b49-b14a-2d21568b5318","Type":"ContainerStarted","Data":"0658bf15bd1dd2ab8141920f3f78c9f190e3063a8a54c9dbec9ef0e850dce0ca"} Nov 03 21:01:33.269078 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:33.269024 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" podStartSLOduration=2.269008486 podStartE2EDuration="2.269008486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:33.268336227 +0000 UTC m=+842.428408500" watchObservedRunningTime="2026-11-03 21:01:33.269008486 +0000 UTC m=+842.429080758" Nov 03 21:01:34.133862 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:34.133829 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:37.583175 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:37.583138 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:37.775795 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:37.775763 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:37.971474 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:37.971436 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:42.134221 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:42.134134 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:42.146829 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:42.146793 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:42.215314 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:42.215283 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:01:42.522606 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:42.522577 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:42.527572 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:01:42.527539 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:01:47.027721 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:47.027678 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:51.499878 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:51.499843 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:52.563239 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:52.563204 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:54.972753 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:54.972713 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:55.164315 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:55.164280 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:56.411400 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:01:56.411362 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:00.920556 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:00.920515 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:01.376642 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:01.376605 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:04.353171 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.353137 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55"] Nov 03 21:02:04.356973 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.356946 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.411561 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.411530 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55"] Nov 03 21:02:04.450447 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.450413 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-home\") pod \"simulator-w0-791c32-kserve-5965f5b9c6-tbf55\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.450447 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.450456 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-dshm\") pod \"simulator-w0-791c32-kserve-5965f5b9c6-tbf55\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.450672 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.450474 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf9455eb-457d-4fec-aca7-f55202209d2c-tls-certs\") pod \"simulator-w0-791c32-kserve-5965f5b9c6-tbf55\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.450672 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.450554 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-tmp-dir\") pod \"simulator-w0-791c32-kserve-5965f5b9c6-tbf55\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.450672 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.450595 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq9bg\" (UniqueName: \"kubernetes.io/projected/cf9455eb-457d-4fec-aca7-f55202209d2c-kube-api-access-gq9bg\") pod \"simulator-w0-791c32-kserve-5965f5b9c6-tbf55\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.450672 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.450647 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-model-cache\") pod \"simulator-w0-791c32-kserve-5965f5b9c6-tbf55\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.456288 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.456262 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-791c32\"/\"default-dockercfg-t5gcp\"" Nov 03 21:02:04.456425 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.456292 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-791c32\"/\"kube-root-ca.crt\"" Nov 03 21:02:04.456425 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.456262 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-791c32\"/\"openshift-service-ca.crt\"" Nov 03 21:02:04.456425 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.456329 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-791c32\"/\"simulator-w0-791c32-kserve-self-signed-certs\"" Nov 03 21:02:04.491378 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.491343 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:04.551607 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.551568 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-home\") pod \"simulator-w0-791c32-kserve-5965f5b9c6-tbf55\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.551815 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.551616 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-dshm\") pod \"simulator-w0-791c32-kserve-5965f5b9c6-tbf55\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.551815 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.551639 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf9455eb-457d-4fec-aca7-f55202209d2c-tls-certs\") pod \"simulator-w0-791c32-kserve-5965f5b9c6-tbf55\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.551815 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.551674 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-tmp-dir\") pod \"simulator-w0-791c32-kserve-5965f5b9c6-tbf55\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.551815 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.551698 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq9bg\" (UniqueName: \"kubernetes.io/projected/cf9455eb-457d-4fec-aca7-f55202209d2c-kube-api-access-gq9bg\") pod \"simulator-w0-791c32-kserve-5965f5b9c6-tbf55\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.551815 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.551721 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-model-cache\") pod \"simulator-w0-791c32-kserve-5965f5b9c6-tbf55\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.552106 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.552050 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-home\") pod \"simulator-w0-791c32-kserve-5965f5b9c6-tbf55\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.552106 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.552091 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-tmp-dir\") pod \"simulator-w0-791c32-kserve-5965f5b9c6-tbf55\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.552214 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.552164 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-model-cache\") pod \"simulator-w0-791c32-kserve-5965f5b9c6-tbf55\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.554016 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.553994 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-dshm\") pod \"simulator-w0-791c32-kserve-5965f5b9c6-tbf55\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.554213 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.554194 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf9455eb-457d-4fec-aca7-f55202209d2c-tls-certs\") pod \"simulator-w0-791c32-kserve-5965f5b9c6-tbf55\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.611052 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.610960 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq9bg\" (UniqueName: \"kubernetes.io/projected/cf9455eb-457d-4fec-aca7-f55202209d2c-kube-api-access-gq9bg\") pod \"simulator-w0-791c32-kserve-5965f5b9c6-tbf55\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:04.669414 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:04.669368 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:05.065291 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:05.065078 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55"] Nov 03 21:02:05.066681 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:02:05.066651 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf9455eb_457d_4fec_aca7_f55202209d2c.slice/crio-a812ebc08aeee073045a89e0f499434114f1ab3cf43c95a288fc9aa6c3357efb WatchSource:0}: Error finding container a812ebc08aeee073045a89e0f499434114f1ab3cf43c95a288fc9aa6c3357efb: Status 404 returned error can't find the container with id a812ebc08aeee073045a89e0f499434114f1ab3cf43c95a288fc9aa6c3357efb Nov 03 21:02:05.323047 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:05.322944 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" event={"ID":"cf9455eb-457d-4fec-aca7-f55202209d2c","Type":"ContainerStarted","Data":"62952a2812750d957e5b0940b6de5bdaf5856b9cec924dd06ce80c9f88b18b4d"} Nov 03 21:02:05.323047 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:05.322998 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" event={"ID":"cf9455eb-457d-4fec-aca7-f55202209d2c","Type":"ContainerStarted","Data":"a812ebc08aeee073045a89e0f499434114f1ab3cf43c95a288fc9aa6c3357efb"} Nov 03 21:02:05.435750 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:05.435695 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" podStartSLOduration=1.435680179 podStartE2EDuration="1.435680179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:05.433778286 +0000 UTC m=+874.593850558" watchObservedRunningTime="2026-11-03 21:02:05.435680179 +0000 UTC m=+874.595752450" Nov 03 21:02:05.509707 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:05.509670 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:05.669742 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:05.669664 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:10.018281 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:10.018244 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:12.528245 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:02:12.528133 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:02:13.384876 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.384841 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x"] Nov 03 21:02:13.389018 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.388994 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.443589 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.443557 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x"] Nov 03 21:02:13.488343 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.488303 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-099a5e\"/\"kube-root-ca.crt\"" Nov 03 21:02:13.488536 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.488418 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-099a5e\"/\"default-dockercfg-dksr6\"" Nov 03 21:02:13.488536 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.488429 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-099a5e\"/\"openshift-service-ca.crt\"" Nov 03 21:02:13.488536 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.488479 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-099a5e\"/\"premium-w2-099a5e-kserve-self-signed-certs\"" Nov 03 21:02:13.523030 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.522986 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:13.540442 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.540411 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-dshm\") pod \"premium-w2-099a5e-kserve-867dccb69-6zx5x\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.540896 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.540493 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-home\") pod \"premium-w2-099a5e-kserve-867dccb69-6zx5x\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.540896 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.540526 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-tmp-dir\") pod \"premium-w2-099a5e-kserve-867dccb69-6zx5x\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.540896 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.540715 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-model-cache\") pod \"premium-w2-099a5e-kserve-867dccb69-6zx5x\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.540896 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.540821 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa8baac2-125d-42f6-8449-5fd590bf4b2b-tls-certs\") pod \"premium-w2-099a5e-kserve-867dccb69-6zx5x\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.540896 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.540856 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb52g\" (UniqueName: \"kubernetes.io/projected/fa8baac2-125d-42f6-8449-5fd590bf4b2b-kube-api-access-sb52g\") pod \"premium-w2-099a5e-kserve-867dccb69-6zx5x\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.642038 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.641938 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-tmp-dir\") pod \"premium-w2-099a5e-kserve-867dccb69-6zx5x\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.642212 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.642081 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-model-cache\") pod \"premium-w2-099a5e-kserve-867dccb69-6zx5x\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.642212 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.642119 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa8baac2-125d-42f6-8449-5fd590bf4b2b-tls-certs\") pod \"premium-w2-099a5e-kserve-867dccb69-6zx5x\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.642212 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.642144 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb52g\" (UniqueName: \"kubernetes.io/projected/fa8baac2-125d-42f6-8449-5fd590bf4b2b-kube-api-access-sb52g\") pod \"premium-w2-099a5e-kserve-867dccb69-6zx5x\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.642212 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.642175 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-dshm\") pod \"premium-w2-099a5e-kserve-867dccb69-6zx5x\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.642424 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.642230 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-home\") pod \"premium-w2-099a5e-kserve-867dccb69-6zx5x\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.642499 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.642472 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-tmp-dir\") pod \"premium-w2-099a5e-kserve-867dccb69-6zx5x\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.642567 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.642522 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-model-cache\") pod \"premium-w2-099a5e-kserve-867dccb69-6zx5x\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.642686 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.642654 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-home\") pod \"premium-w2-099a5e-kserve-867dccb69-6zx5x\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.644476 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.644451 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-dshm\") pod \"premium-w2-099a5e-kserve-867dccb69-6zx5x\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.644643 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.644625 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa8baac2-125d-42f6-8449-5fd590bf4b2b-tls-certs\") pod \"premium-w2-099a5e-kserve-867dccb69-6zx5x\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.700959 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.700897 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb52g\" (UniqueName: \"kubernetes.io/projected/fa8baac2-125d-42f6-8449-5fd590bf4b2b-kube-api-access-sb52g\") pod \"premium-w2-099a5e-kserve-867dccb69-6zx5x\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:13.701663 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:13.701638 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:14.087815 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:14.087772 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x"] Nov 03 21:02:14.093888 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:02:14.093857 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa8baac2_125d_42f6_8449_5fd590bf4b2b.slice/crio-d0fb90122402c6845d6b000e2bdbd1d091408617d37e2ac9119ed5ef9a3c098d WatchSource:0}: Error finding container d0fb90122402c6845d6b000e2bdbd1d091408617d37e2ac9119ed5ef9a3c098d: Status 404 returned error can't find the container with id d0fb90122402c6845d6b000e2bdbd1d091408617d37e2ac9119ed5ef9a3c098d Nov 03 21:02:14.368949 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:14.368846 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" event={"ID":"fa8baac2-125d-42f6-8449-5fd590bf4b2b","Type":"ContainerStarted","Data":"75d48a7aa0c9b43e9d4131786a80a1c965eb47dc57619b8399d71bed70b6c530"} Nov 03 21:02:14.368949 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:14.368887 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" event={"ID":"fa8baac2-125d-42f6-8449-5fd590bf4b2b","Type":"ContainerStarted","Data":"d0fb90122402c6845d6b000e2bdbd1d091408617d37e2ac9119ed5ef9a3c098d"} Nov 03 21:02:14.480819 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:14.480762 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" podStartSLOduration=1.480747156 podStartE2EDuration="1.480747156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:14.479975535 +0000 UTC m=+883.640047807" watchObservedRunningTime="2026-11-03 21:02:14.480747156 +0000 UTC m=+883.640819428" Nov 03 21:02:14.515664 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:14.515621 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:14.669861 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:14.669775 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:14.682461 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:14.682432 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:14.702634 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:14.702600 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:15.385443 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:15.385415 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:02:19.102370 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:19.102336 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:23.702552 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:23.702519 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:23.715486 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:23.715458 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:23.754259 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:23.754224 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:24.434298 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:24.434264 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 21:02:26.015365 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:26.015333 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:28.214309 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:28.214269 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:29.978516 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:29.978484 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:31.044621 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:31.044590 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:02:31.048931 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:31.048884 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:02:32.829307 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:32.829269 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:35.315552 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:35.315516 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:37.453410 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:37.453354 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:42.025022 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:42.024986 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:42.528351 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:02:42.528317 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:02:46.651856 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:46.651816 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:51.312771 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:51.312735 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:55.865552 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:02:55.865510 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:00.473659 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:00.473625 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:04.947535 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:04.947497 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:08.345314 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:08.345265 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:09.669482 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:09.669445 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:12.529254 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:03:12.529156 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:03:14.378168 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:14.378121 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:18.999927 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:18.999862 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:23.653747 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:23.653708 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:26.343415 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:26.343371 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:28.224066 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:28.224028 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:32.726736 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:32.726688 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:33.847153 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:33.847117 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb"] Nov 03 21:03:33.851016 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:33.850993 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:33.916042 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:33.916004 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-dshm\") pod \"premium-w1-a41ad0-kserve-756f497796-4tttb\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:33.916239 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:33.916127 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-tmp-dir\") pod \"premium-w1-a41ad0-kserve-756f497796-4tttb\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:33.916239 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:33.916211 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/662bceeb-beae-4a4f-b08a-08706bed2d2e-tls-certs\") pod \"premium-w1-a41ad0-kserve-756f497796-4tttb\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:33.916347 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:33.916314 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-home\") pod \"premium-w1-a41ad0-kserve-756f497796-4tttb\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:33.916409 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:33.916392 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98p6z\" (UniqueName: \"kubernetes.io/projected/662bceeb-beae-4a4f-b08a-08706bed2d2e-kube-api-access-98p6z\") pod \"premium-w1-a41ad0-kserve-756f497796-4tttb\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:33.916452 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:33.916432 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-model-cache\") pod \"premium-w1-a41ad0-kserve-756f497796-4tttb\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:33.950256 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:33.950222 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a41ad0\"/\"openshift-service-ca.crt\"" Nov 03 21:03:33.950256 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:33.950235 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a41ad0\"/\"kube-root-ca.crt\"" Nov 03 21:03:33.950570 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:33.950548 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a41ad0\"/\"premium-w1-a41ad0-kserve-self-signed-certs\"" Nov 03 21:03:33.950647 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:33.950632 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a41ad0\"/\"default-dockercfg-zqb9q\"" Nov 03 21:03:33.952829 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:33.952807 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb"] Nov 03 21:03:33.980549 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:33.980516 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:34.017649 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:34.017614 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98p6z\" (UniqueName: \"kubernetes.io/projected/662bceeb-beae-4a4f-b08a-08706bed2d2e-kube-api-access-98p6z\") pod \"premium-w1-a41ad0-kserve-756f497796-4tttb\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:34.017869 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:34.017673 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-model-cache\") pod \"premium-w1-a41ad0-kserve-756f497796-4tttb\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:34.017869 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:34.017738 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-dshm\") pod \"premium-w1-a41ad0-kserve-756f497796-4tttb\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:34.018023 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:34.017969 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-tmp-dir\") pod \"premium-w1-a41ad0-kserve-756f497796-4tttb\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:34.018230 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:34.018184 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-model-cache\") pod \"premium-w1-a41ad0-kserve-756f497796-4tttb\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:34.018348 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:34.018235 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-tmp-dir\") pod \"premium-w1-a41ad0-kserve-756f497796-4tttb\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:34.018442 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:34.018350 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/662bceeb-beae-4a4f-b08a-08706bed2d2e-tls-certs\") pod \"premium-w1-a41ad0-kserve-756f497796-4tttb\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:34.018564 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:34.018545 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-home\") pod \"premium-w1-a41ad0-kserve-756f497796-4tttb\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:34.018825 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:34.018805 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-home\") pod \"premium-w1-a41ad0-kserve-756f497796-4tttb\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:34.020324 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:34.020293 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-dshm\") pod \"premium-w1-a41ad0-kserve-756f497796-4tttb\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:34.020742 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:34.020721 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/662bceeb-beae-4a4f-b08a-08706bed2d2e-tls-certs\") pod \"premium-w1-a41ad0-kserve-756f497796-4tttb\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:34.076923 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:34.076879 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98p6z\" (UniqueName: \"kubernetes.io/projected/662bceeb-beae-4a4f-b08a-08706bed2d2e-kube-api-access-98p6z\") pod \"premium-w1-a41ad0-kserve-756f497796-4tttb\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:34.171966 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:34.171852 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:34.574235 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:34.574198 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb"] Nov 03 21:03:34.579953 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:03:34.579922 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod662bceeb_beae_4a4f_b08a_08706bed2d2e.slice/crio-3d53d4987aebb33118e8a4cb112925c83c1bf1ee6edd7c9bb1a25e0581ee64a8 WatchSource:0}: Error finding container 3d53d4987aebb33118e8a4cb112925c83c1bf1ee6edd7c9bb1a25e0581ee64a8: Status 404 returned error can't find the container with id 3d53d4987aebb33118e8a4cb112925c83c1bf1ee6edd7c9bb1a25e0581ee64a8 Nov 03 21:03:34.802477 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:34.802438 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" event={"ID":"662bceeb-beae-4a4f-b08a-08706bed2d2e","Type":"ContainerStarted","Data":"59ac55ed0c753f0d0d5252a44940ff4fd633caa838c1a0508918a3c56b259ab7"} Nov 03 21:03:34.802656 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:34.802491 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" event={"ID":"662bceeb-beae-4a4f-b08a-08706bed2d2e","Type":"ContainerStarted","Data":"3d53d4987aebb33118e8a4cb112925c83c1bf1ee6edd7c9bb1a25e0581ee64a8"} Nov 03 21:03:34.915187 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:34.915076 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" podStartSLOduration=1.915061314 podStartE2EDuration="1.915061314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:34.913782047 +0000 UTC m=+964.073854322" watchObservedRunningTime="2026-11-03 21:03:34.915061314 +0000 UTC m=+964.075133585" Nov 03 21:03:35.172207 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:35.172112 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:37.313770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:37.313737 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:41.814148 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:41.814116 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:42.527534 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:03:42.527489 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:03:44.172855 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:44.172802 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:44.192436 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:44.192399 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:44.870658 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:44.870624 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 21:03:46.327506 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:46.327471 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:50.852711 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:50.852662 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:55.422636 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:55.422600 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:59.949705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:03:59.949668 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:04.502643 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:04.502613 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:09.034354 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:09.034312 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:12.527927 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:04:12.527879 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:04:13.509437 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:13.509398 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:18.017621 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:18.017580 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:18.271155 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:18.271070 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:18.452724 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:18.452679 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:18.640009 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:18.639895 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:22.063397 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:22.063360 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:22.617613 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:22.617584 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:26.686583 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:26.686545 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:27.223321 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:27.223252 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:31.714625 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:31.714580 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:36.279438 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:36.279402 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:38.512436 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.512401 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f"] Nov 03 21:04:38.516521 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.516497 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:38.616461 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.616411 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-099a5e\"/\"e2e-conflict-a-3ba3a28c-kserve-self-signed-certs\"" Nov 03 21:04:38.618886 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.618858 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f"] Nov 03 21:04:38.640498 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.640462 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:38.706969 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.706920 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-home\") pod \"e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:38.707160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.707066 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmh97\" (UniqueName: \"kubernetes.io/projected/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-kube-api-access-pmh97\") pod \"e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:38.707160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.707112 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-dshm\") pod \"e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:38.707160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.707136 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-model-cache\") pod \"e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:38.707402 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.707276 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-tls-certs\") pod \"e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:38.707402 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.707331 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-tmp-dir\") pod \"e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:38.808049 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.807950 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-tls-certs\") pod \"e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:38.808049 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.808015 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-tmp-dir\") pod \"e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:38.808300 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.808091 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-home\") pod \"e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:38.808300 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.808195 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmh97\" (UniqueName: \"kubernetes.io/projected/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-kube-api-access-pmh97\") pod \"e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:38.808300 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.808244 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-dshm\") pod \"e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:38.808300 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.808268 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-model-cache\") pod \"e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:38.808545 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.808516 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-tmp-dir\") pod \"e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:38.808719 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.808695 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-model-cache\") pod \"e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:38.808791 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.808530 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-home\") pod \"e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:38.811098 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.811067 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-tls-certs\") pod \"e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:38.811214 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.811108 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-dshm\") pod \"e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:38.866963 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:38.866923 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmh97\" (UniqueName: \"kubernetes.io/projected/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-kube-api-access-pmh97\") pod \"e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:39.129409 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:39.129311 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:39.505940 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:39.505896 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f"] Nov 03 21:04:39.512206 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:04:39.512176 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c19e318_7e17_4a5c_8fa5_eced96a4f0fc.slice/crio-dafce474f073a797b6986a3715fc43a4ce1b8db9398dc827876a2beea8bf6f58 WatchSource:0}: Error finding container dafce474f073a797b6986a3715fc43a4ce1b8db9398dc827876a2beea8bf6f58: Status 404 returned error can't find the container with id dafce474f073a797b6986a3715fc43a4ce1b8db9398dc827876a2beea8bf6f58 Nov 03 21:04:40.164052 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:40.163969 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" event={"ID":"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc","Type":"ContainerStarted","Data":"15203fd312415fca8ac1dcab5a2e24c1ceb2df4839061f4ad294011e235787f3"} Nov 03 21:04:40.164052 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:40.164017 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" event={"ID":"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc","Type":"ContainerStarted","Data":"dafce474f073a797b6986a3715fc43a4ce1b8db9398dc827876a2beea8bf6f58"} Nov 03 21:04:40.276559 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:40.276507 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" podStartSLOduration=2.276493545 podStartE2EDuration="2.276493545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:40.27521125 +0000 UTC m=+1029.435283521" watchObservedRunningTime="2026-11-03 21:04:40.276493545 +0000 UTC m=+1029.436565825" Nov 03 21:04:40.821359 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:40.821327 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:41.129938 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:41.129837 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:42.631895 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:04:42.631840 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:04:44.080263 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.080229 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9"] Nov 03 21:04:44.084603 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.084576 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.172648 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.172614 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.172823 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.172727 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.172823 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.172757 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.172823 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.172779 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.172823 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.172794 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.173033 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.172893 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.173033 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.172980 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.173118 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.173030 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrxmq\" (UniqueName: \"kubernetes.io/projected/a175a685-3ef1-4d0e-ac80-ad2572d87bde-kube-api-access-vrxmq\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.173118 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.173106 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.177550 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.177518 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9"] Nov 03 21:04:44.273802 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.273765 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.274035 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.273841 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.274035 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.273874 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.274035 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.273893 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.274035 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.273968 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.274035 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.274005 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.274035 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.274033 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.274329 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.274081 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrxmq\" (UniqueName: \"kubernetes.io/projected/a175a685-3ef1-4d0e-ac80-ad2572d87bde-kube-api-access-vrxmq\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.274329 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.274116 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.274329 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.274290 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.274492 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.274415 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.274550 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.274485 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.274602 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.274559 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.274756 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.274737 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.276329 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.276309 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.276441 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.276425 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.334025 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.333939 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrxmq\" (UniqueName: \"kubernetes.io/projected/a175a685-3ef1-4d0e-ac80-ad2572d87bde-kube-api-access-vrxmq\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.337719 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.337686 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-q4hj9\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.399968 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.399929 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:44.847118 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.847084 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9"] Nov 03 21:04:44.852529 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:04:44.852498 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda175a685_3ef1_4d0e_ac80_ad2572d87bde.slice/crio-dcdf9d570917f9e63e607c756d3226a1d4eab235fb0640e2aaec6b6db7565796 WatchSource:0}: Error finding container dcdf9d570917f9e63e607c756d3226a1d4eab235fb0640e2aaec6b6db7565796: Status 404 returned error can't find the container with id dcdf9d570917f9e63e607c756d3226a1d4eab235fb0640e2aaec6b6db7565796 Nov 03 21:04:44.854845 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.854811 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:04:44.854979 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.854881 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:04:44.854979 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:44.854942 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:04:45.206855 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:45.206803 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" event={"ID":"a175a685-3ef1-4d0e-ac80-ad2572d87bde","Type":"ContainerStarted","Data":"d95f78a525895e6c50f051243e2a23c96885b9e449d17277a9c8274658e95791"} Nov 03 21:04:45.207396 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:45.206874 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" event={"ID":"a175a685-3ef1-4d0e-ac80-ad2572d87bde","Type":"ContainerStarted","Data":"dcdf9d570917f9e63e607c756d3226a1d4eab235fb0640e2aaec6b6db7565796"} Nov 03 21:04:45.324537 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:45.324480 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" podStartSLOduration=1.324462977 podStartE2EDuration="1.324462977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:45.322403725 +0000 UTC m=+1034.482475998" watchObservedRunningTime="2026-11-03 21:04:45.324462977 +0000 UTC m=+1034.484535248" Nov 03 21:04:45.400959 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:45.400928 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:45.402376 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:45.402353 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:04:45.402435 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:45.402403 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" podUID="a175a685-3ef1-4d0e-ac80-ad2572d87bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 21:04:45.498595 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:45.498509 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:46.401928 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:46.401037 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:04:46.401928 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:46.401119 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" podUID="a175a685-3ef1-4d0e-ac80-ad2572d87bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 21:04:47.403450 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:47.403408 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:04:47.404105 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:47.404074 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" podUID="a175a685-3ef1-4d0e-ac80-ad2572d87bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 21:04:48.405932 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:48.401329 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:04:48.405932 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:48.401419 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" podUID="a175a685-3ef1-4d0e-ac80-ad2572d87bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 21:04:49.129559 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:49.129517 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:49.142266 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:49.142233 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:49.240803 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:49.240772 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:04:49.412456 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:49.412364 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:04:49.412857 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:49.412448 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" podUID="a175a685-3ef1-4d0e-ac80-ad2572d87bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 21:04:50.123814 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:50.123778 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:50.405988 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:50.405131 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:04:50.405988 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:50.405191 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" podUID="a175a685-3ef1-4d0e-ac80-ad2572d87bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 21:04:51.400555 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:51.400519 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:04:51.400956 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:51.400584 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" podUID="a175a685-3ef1-4d0e-ac80-ad2572d87bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 21:04:52.401073 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:52.401033 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:04:52.401526 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:52.401101 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" podUID="a175a685-3ef1-4d0e-ac80-ad2572d87bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 21:04:53.803084 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:53.803048 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:53.803584 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:53.803437 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:53.804341 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:53.804316 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:04:54.615494 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:54.615460 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:59.106610 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.106574 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:59.201859 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.201822 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6"] Nov 03 21:04:59.205868 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.205839 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.243524 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.243491 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdhwr\" (UniqueName: \"kubernetes.io/projected/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-kube-api-access-fdhwr\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.243705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.243536 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.243705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.243591 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.243705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.243615 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.243705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.243672 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.243705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.243703 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.243914 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.243805 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.243914 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.243822 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.243914 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.243881 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.345631 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.345586 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdhwr\" (UniqueName: \"kubernetes.io/projected/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-kube-api-access-fdhwr\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.345851 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.345666 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.345851 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.345700 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.345851 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.345736 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.345851 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.345802 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.345851 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.345849 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.346240 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.345886 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.346240 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.345927 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.346240 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.345974 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.346500 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.346296 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.346500 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.346343 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.346631 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.346599 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.347369 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.347344 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.347740 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.347723 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.348640 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.348621 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.349327 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.349309 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.406472 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.404183 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.406472 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.404266 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdhwr\" (UniqueName: \"kubernetes.io/projected/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-kube-api-access-fdhwr\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-67kl6\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.409396 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.409284 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6"] Nov 03 21:04:59.520391 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.520351 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:04:59.962345 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.962304 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6"] Nov 03 21:04:59.967969 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:04:59.967939 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed228cb_abae_423f_b7f2_3dd6f8a1096f.slice/crio-342270b7857e62b676c42ae581a309923ecb1af4d8d17d7ed073898d7b378688 WatchSource:0}: Error finding container 342270b7857e62b676c42ae581a309923ecb1af4d8d17d7ed073898d7b378688: Status 404 returned error can't find the container with id 342270b7857e62b676c42ae581a309923ecb1af4d8d17d7ed073898d7b378688 Nov 03 21:04:59.971058 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.971023 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:04:59.971183 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.971106 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:04:59.971183 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:04:59.971150 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:05:00.304581 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:00.303763 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" event={"ID":"6ed228cb-abae-423f-b7f2-3dd6f8a1096f","Type":"ContainerStarted","Data":"c122f8eb142ff2a52e9eb2e6908940a578504cdb3a9ba3178b21a30a0b6e0deb"} Nov 03 21:05:00.304581 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:00.303833 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" event={"ID":"6ed228cb-abae-423f-b7f2-3dd6f8a1096f","Type":"ContainerStarted","Data":"342270b7857e62b676c42ae581a309923ecb1af4d8d17d7ed073898d7b378688"} Nov 03 21:05:00.431424 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:00.431362 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" podStartSLOduration=1.431341217 podStartE2EDuration="1.431341217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:00.430327396 +0000 UTC m=+1049.590399679" watchObservedRunningTime="2026-11-03 21:05:00.431341217 +0000 UTC m=+1049.591413490" Nov 03 21:05:00.524062 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:00.524024 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:05:01.529929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.524239 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-67kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 21:05:01.529929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.524324 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" podUID="6ed228cb-abae-423f-b7f2-3dd6f8a1096f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 21:05:01.759756 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.759394 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll"] Nov 03 21:05:01.764618 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.764592 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.824002 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.816060 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-099a5e-openshift-default-dockercfg-2h6nd\"" Nov 03 21:05:01.868172 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.867957 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll"] Nov 03 21:05:01.876979 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.876944 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.877160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.877008 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.877160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.877100 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.877160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.877142 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.877326 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.877166 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdvj9\" (UniqueName: \"kubernetes.io/projected/30c75fe2-5f87-4852-9f1f-703f4ff01304-kube-api-access-vdvj9\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.877326 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.877291 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.877435 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.877341 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.877435 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.877383 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.877526 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.877501 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c75fe2-5f87-4852-9f1f-703f4ff01304-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.978772 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.978733 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.979010 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.978785 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c75fe2-5f87-4852-9f1f-703f4ff01304-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.979010 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.978843 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.979010 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.978927 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.979010 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.978972 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.979235 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.979012 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.979235 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.979037 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdvj9\" (UniqueName: \"kubernetes.io/projected/30c75fe2-5f87-4852-9f1f-703f4ff01304-kube-api-access-vdvj9\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.979235 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.979088 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.979235 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.979126 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.979446 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.979328 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.979510 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.979459 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.979742 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.979682 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c75fe2-5f87-4852-9f1f-703f4ff01304-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.979876 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.979825 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:01.979963 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:01.979897 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:02.016621 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:02.005262 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:02.019941 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:02.018600 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:02.040923 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:02.039814 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:02.040923 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:02.040826 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdvj9\" (UniqueName: \"kubernetes.io/projected/30c75fe2-5f87-4852-9f1f-703f4ff01304-kube-api-access-vdvj9\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:02.079078 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:02.078977 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:02.521592 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:02.521553 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-67kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 21:05:02.521785 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:02.521610 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" podUID="6ed228cb-abae-423f-b7f2-3dd6f8a1096f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 21:05:02.578608 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:02.578575 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll"] Nov 03 21:05:02.606565 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:05:02.606530 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c75fe2_5f87_4852_9f1f_703f4ff01304.slice/crio-a42392eae3b54db16e3547aeac7b0e585bada8bb6a6716cdacb7a79b8d3df7c7 WatchSource:0}: Error finding container a42392eae3b54db16e3547aeac7b0e585bada8bb6a6716cdacb7a79b8d3df7c7: Status 404 returned error can't find the container with id a42392eae3b54db16e3547aeac7b0e585bada8bb6a6716cdacb7a79b8d3df7c7 Nov 03 21:05:02.609954 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:02.609921 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:05:02.610044 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:02.609990 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:05:02.610044 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:02.610023 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:05:03.367444 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:03.367406 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" event={"ID":"30c75fe2-5f87-4852-9f1f-703f4ff01304","Type":"ContainerStarted","Data":"a667b2f37c93fa0ef914966a11a326318626447b55621b9e0f5ecb599850ce65"} Nov 03 21:05:03.367611 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:03.367460 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" event={"ID":"30c75fe2-5f87-4852-9f1f-703f4ff01304","Type":"ContainerStarted","Data":"a42392eae3b54db16e3547aeac7b0e585bada8bb6a6716cdacb7a79b8d3df7c7"} Nov 03 21:05:03.530929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:03.528099 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-67kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 21:05:03.530929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:03.528165 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" podUID="6ed228cb-abae-423f-b7f2-3dd6f8a1096f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 21:05:03.534386 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:03.534319 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" podStartSLOduration=2.534299725 podStartE2EDuration="2.534299725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:03.533043855 +0000 UTC m=+1052.693116156" watchObservedRunningTime="2026-11-03 21:05:03.534299725 +0000 UTC m=+1052.694371998" Nov 03 21:05:03.771774 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:03.771735 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:04.050525 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.050424 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln"] Nov 03 21:05:04.054376 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.054347 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.079206 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.079178 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:04.113929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.098120 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 21:05:04.113929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.098202 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" podUID="30c75fe2-5f87-4852-9f1f-703f4ff01304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 21:05:04.113929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.104995 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-791c32-openshift-default-dockercfg-zfm58\"" Nov 03 21:05:04.138276 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.138241 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.138456 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.138360 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.138456 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.138401 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.138527 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.138511 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.138563 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.138554 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z5bx\" (UniqueName: \"kubernetes.io/projected/bafd62fa-db24-4cbf-ad55-1495f7daef2f-kube-api-access-9z5bx\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.138622 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.138606 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.138746 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.138730 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.138841 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.138828 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.138933 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.138921 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.156649 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.156618 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln"] Nov 03 21:05:04.243108 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.243069 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.243312 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.243137 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.243312 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.243222 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.243312 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.243258 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z5bx\" (UniqueName: \"kubernetes.io/projected/bafd62fa-db24-4cbf-ad55-1495f7daef2f-kube-api-access-9z5bx\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.243481 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.243316 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.243481 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.243367 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.243481 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.243396 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.243481 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.243427 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.243673 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.243490 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.244010 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.243981 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.244350 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.244328 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.245606 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.245579 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.245858 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.245839 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.246513 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.246475 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.247584 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.247561 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.255468 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.255445 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.332975 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.332868 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.332975 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.332884 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z5bx\" (UniqueName: \"kubernetes.io/projected/bafd62fa-db24-4cbf-ad55-1495f7daef2f-kube-api-access-9z5bx\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.371632 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.371599 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:04.532484 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.532434 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-67kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 21:05:04.532692 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.532499 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" podUID="6ed228cb-abae-423f-b7f2-3dd6f8a1096f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 21:05:04.933355 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.931563 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln"] Nov 03 21:05:04.935400 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:05:04.935367 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbafd62fa_db24_4cbf_ad55_1495f7daef2f.slice/crio-0f1feb114da736f44388ebb41486f0d65c15175f9fa96a6613a341e3583f65a4 WatchSource:0}: Error finding container 0f1feb114da736f44388ebb41486f0d65c15175f9fa96a6613a341e3583f65a4: Status 404 returned error can't find the container with id 0f1feb114da736f44388ebb41486f0d65c15175f9fa96a6613a341e3583f65a4 Nov 03 21:05:04.938768 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.937806 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:05:04.938768 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.937899 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:05:04.938768 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:04.937960 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:05:05.080503 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:05.080470 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 21:05:05.080662 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:05.080525 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" podUID="30c75fe2-5f87-4852-9f1f-703f4ff01304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 21:05:05.389164 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:05.382835 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" event={"ID":"bafd62fa-db24-4cbf-ad55-1495f7daef2f","Type":"ContainerStarted","Data":"5f8a0bfa1249a30f6d072da4237f771f08582283a9d258509ac90d89781c33d3"} Nov 03 21:05:05.399140 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:05.399056 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" event={"ID":"bafd62fa-db24-4cbf-ad55-1495f7daef2f","Type":"ContainerStarted","Data":"0f1feb114da736f44388ebb41486f0d65c15175f9fa96a6613a341e3583f65a4"} Nov 03 21:05:05.521158 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:05.521110 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-67kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 21:05:05.521465 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:05.521436 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" podUID="6ed228cb-abae-423f-b7f2-3dd6f8a1096f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 21:05:06.079471 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:06.079438 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 21:05:06.079848 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:06.079496 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" podUID="30c75fe2-5f87-4852-9f1f-703f4ff01304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 21:05:06.372765 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:06.372660 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:06.374228 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:06.374203 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:05:06.374326 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:06.374255 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" podUID="bafd62fa-db24-4cbf-ad55-1495f7daef2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 21:05:06.528428 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:06.528376 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-67kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 21:05:06.528621 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:06.528443 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" podUID="6ed228cb-abae-423f-b7f2-3dd6f8a1096f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 21:05:07.079416 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:07.079380 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 21:05:07.079628 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:07.079445 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" podUID="30c75fe2-5f87-4852-9f1f-703f4ff01304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 21:05:07.372648 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:07.372551 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:05:07.372648 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:07.372618 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" podUID="bafd62fa-db24-4cbf-ad55-1495f7daef2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 21:05:07.521352 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:07.521313 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-67kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 21:05:07.521646 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:07.521613 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" podUID="6ed228cb-abae-423f-b7f2-3dd6f8a1096f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 21:05:08.080268 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:08.080228 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 21:05:08.080674 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:08.080292 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" podUID="30c75fe2-5f87-4852-9f1f-703f4ff01304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 21:05:08.373130 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:08.373036 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:05:08.373130 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:08.373090 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" podUID="bafd62fa-db24-4cbf-ad55-1495f7daef2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 21:05:08.524804 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:08.524338 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-67kl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 21:05:08.524804 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:08.524401 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" podUID="6ed228cb-abae-423f-b7f2-3dd6f8a1096f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 21:05:08.714931 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:08.712510 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:09.079953 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:09.079843 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 21:05:09.079953 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:09.079931 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" podUID="30c75fe2-5f87-4852-9f1f-703f4ff01304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 21:05:09.372924 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:09.372766 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:05:09.372924 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:09.372847 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" podUID="bafd62fa-db24-4cbf-ad55-1495f7daef2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 21:05:09.819015 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:09.818971 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:05:09.820127 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:09.820104 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:05:09.821237 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:09.821217 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:05:10.058303 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:10.058249 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" podStartSLOduration=6.058234372 podStartE2EDuration="6.058234372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:05.520096801 +0000 UTC m=+1054.680169074" watchObservedRunningTime="2026-11-03 21:05:10.058234372 +0000 UTC m=+1059.218306651" Nov 03 21:05:10.301928 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:10.301234 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:10.305926 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:10.302430 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:10.311482 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:10.311448 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:05:10.373019 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:10.372982 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:05:10.373457 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:10.373033 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" podUID="bafd62fa-db24-4cbf-ad55-1495f7daef2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 21:05:11.373210 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:11.373178 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:05:11.373716 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:11.373231 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" podUID="bafd62fa-db24-4cbf-ad55-1495f7daef2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 21:05:11.830422 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:11.830388 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:12.377086 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:12.377054 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:12.377520 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:12.377447 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:12.378160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:12.378143 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:05:12.527888 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:05:12.527850 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:05:13.305674 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:13.305636 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:17.892194 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:17.892161 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:22.405518 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:22.405469 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:26.931623 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:26.931581 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:27.118399 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:27.118300 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:31.413816 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:31.413776 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:32.717927 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.717875 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9"] Nov 03 21:05:32.722173 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.722151 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:32.821611 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.821581 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-791c32\"/\"distinct-w0-791c32-kserve-self-signed-certs\"" Nov 03 21:05:32.824252 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.824231 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9"] Nov 03 21:05:32.855155 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.855088 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:32.896269 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.896226 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h99jc\" (UniqueName: \"kubernetes.io/projected/b48999ea-6a3b-427c-a22b-621a5cc997db-kube-api-access-h99jc\") pod \"distinct-w0-791c32-kserve-78c78966dd-mmzm9\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:32.896488 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.896365 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-model-cache\") pod \"distinct-w0-791c32-kserve-78c78966dd-mmzm9\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:32.896488 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.896427 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b48999ea-6a3b-427c-a22b-621a5cc997db-tls-certs\") pod \"distinct-w0-791c32-kserve-78c78966dd-mmzm9\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:32.896616 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.896487 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-tmp-dir\") pod \"distinct-w0-791c32-kserve-78c78966dd-mmzm9\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:32.896616 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.896542 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-dshm\") pod \"distinct-w0-791c32-kserve-78c78966dd-mmzm9\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:32.896616 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.896600 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-home\") pod \"distinct-w0-791c32-kserve-78c78966dd-mmzm9\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:32.997531 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.997444 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h99jc\" (UniqueName: \"kubernetes.io/projected/b48999ea-6a3b-427c-a22b-621a5cc997db-kube-api-access-h99jc\") pod \"distinct-w0-791c32-kserve-78c78966dd-mmzm9\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:32.997531 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.997494 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-model-cache\") pod \"distinct-w0-791c32-kserve-78c78966dd-mmzm9\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:32.997531 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.997521 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b48999ea-6a3b-427c-a22b-621a5cc997db-tls-certs\") pod \"distinct-w0-791c32-kserve-78c78966dd-mmzm9\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:32.997826 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.997566 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-tmp-dir\") pod \"distinct-w0-791c32-kserve-78c78966dd-mmzm9\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:32.997826 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.997609 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-dshm\") pod \"distinct-w0-791c32-kserve-78c78966dd-mmzm9\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:32.997826 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.997657 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-home\") pod \"distinct-w0-791c32-kserve-78c78966dd-mmzm9\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:32.997994 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.997971 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-model-cache\") pod \"distinct-w0-791c32-kserve-78c78966dd-mmzm9\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:32.998069 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.998043 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-tmp-dir\") pod \"distinct-w0-791c32-kserve-78c78966dd-mmzm9\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:32.998174 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:32.998154 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-home\") pod \"distinct-w0-791c32-kserve-78c78966dd-mmzm9\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:33.000116 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:33.000091 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-dshm\") pod \"distinct-w0-791c32-kserve-78c78966dd-mmzm9\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:33.000756 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:33.000736 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b48999ea-6a3b-427c-a22b-621a5cc997db-tls-certs\") pod \"distinct-w0-791c32-kserve-78c78966dd-mmzm9\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:33.055777 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:33.055747 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h99jc\" (UniqueName: \"kubernetes.io/projected/b48999ea-6a3b-427c-a22b-621a5cc997db-kube-api-access-h99jc\") pod \"distinct-w0-791c32-kserve-78c78966dd-mmzm9\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:33.334842 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:33.334762 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:33.709367 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:33.709332 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9"] Nov 03 21:05:33.715521 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:05:33.715492 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb48999ea_6a3b_427c_a22b_621a5cc997db.slice/crio-55d3c95afb9d592afe7693d7229881fe7042bb186797a8beee0d7ac33b00793d WatchSource:0}: Error finding container 55d3c95afb9d592afe7693d7229881fe7042bb186797a8beee0d7ac33b00793d: Status 404 returned error can't find the container with id 55d3c95afb9d592afe7693d7229881fe7042bb186797a8beee0d7ac33b00793d Nov 03 21:05:34.589432 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:34.589388 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" event={"ID":"b48999ea-6a3b-427c-a22b-621a5cc997db","Type":"ContainerStarted","Data":"9ad65fc1ac7e589fd7b5fa327835326ba809b6c7d0df78e390b21a087c37b34c"} Nov 03 21:05:34.589951 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:34.589438 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" event={"ID":"b48999ea-6a3b-427c-a22b-621a5cc997db","Type":"ContainerStarted","Data":"55d3c95afb9d592afe7693d7229881fe7042bb186797a8beee0d7ac33b00793d"} Nov 03 21:05:34.702419 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:34.702364 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" podStartSLOduration=2.702350277 podStartE2EDuration="2.702350277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:34.700105406 +0000 UTC m=+1083.860177678" watchObservedRunningTime="2026-11-03 21:05:34.702350277 +0000 UTC m=+1083.862422549" Nov 03 21:05:35.335519 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:35.335478 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:35.914803 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:35.914769 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:40.416178 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:40.416121 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:42.532260 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:05:42.532218 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:05:43.335510 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:43.335475 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:43.348583 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:43.348549 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:43.654000 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:43.653893 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 21:05:44.998186 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:44.998150 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:49.495849 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:49.495810 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:51.402922 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:51.402869 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:54.014576 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:54.014540 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:58.505998 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:05:58.505966 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:03.161196 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:03.161156 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:04.571953 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:04.571915 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:04.758951 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:04.757639 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:04.950684 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:04.950610 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:06.060927 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:06.060543 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:08.310556 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:08.310509 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:12.528709 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:06:12.528627 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:06:12.909893 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:12.909785 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:17.719925 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:17.719373 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:21.784895 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.784859 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx"] Nov 03 21:06:21.788734 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.788713 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:21.844341 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.844306 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-dshm\") pod \"distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:21.844510 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.844366 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-tls-certs\") pod \"distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:21.844510 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.844473 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-tmp-dir\") pod \"distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:21.844601 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.844561 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6ksh\" (UniqueName: \"kubernetes.io/projected/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-kube-api-access-x6ksh\") pod \"distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:21.844651 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.844633 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-home\") pod \"distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:21.844755 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.844740 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-model-cache\") pod \"distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:21.885955 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.885927 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a41ad0\"/\"distinct-2-w1-a41ad0-kserve-self-signed-certs\"" Nov 03 21:06:21.889796 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.889769 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx"] Nov 03 21:06:21.910520 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.910488 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:21.947411 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.946533 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ksh\" (UniqueName: \"kubernetes.io/projected/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-kube-api-access-x6ksh\") pod \"distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:21.947411 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.946670 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-home\") pod \"distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:21.947411 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.946759 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-model-cache\") pod \"distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:21.947411 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.946805 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-dshm\") pod \"distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:21.947411 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.946859 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-tls-certs\") pod \"distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:21.947411 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.946940 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-tmp-dir\") pod \"distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:21.947411 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.947260 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-home\") pod \"distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:21.947411 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.947287 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-model-cache\") pod \"distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:21.947411 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.947352 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-tmp-dir\") pod \"distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:21.949505 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.949481 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-tls-certs\") pod \"distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:21.949689 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:21.949672 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-dshm\") pod \"distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:22.004997 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:22.004959 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6ksh\" (UniqueName: \"kubernetes.io/projected/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-kube-api-access-x6ksh\") pod \"distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:22.103497 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:22.103389 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:22.216689 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:22.216601 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:22.490249 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:22.490217 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx"] Nov 03 21:06:22.495406 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:06:22.495371 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bdfcd76_3dd3_46fd_a382_0e40e648aa17.slice/crio-7fac3fbd8fce1fc2bd3d39b8550e854eb7b87d44837369246a1bebd15d11fce0 WatchSource:0}: Error finding container 7fac3fbd8fce1fc2bd3d39b8550e854eb7b87d44837369246a1bebd15d11fce0: Status 404 returned error can't find the container with id 7fac3fbd8fce1fc2bd3d39b8550e854eb7b87d44837369246a1bebd15d11fce0 Nov 03 21:06:22.889591 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:22.889502 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" event={"ID":"7bdfcd76-3dd3-46fd-a382-0e40e648aa17","Type":"ContainerStarted","Data":"fdd44493093f157a65556a76d1127e2789c257d4b59ef446f464121da90df6af"} Nov 03 21:06:22.889591 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:22.889552 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" event={"ID":"7bdfcd76-3dd3-46fd-a382-0e40e648aa17","Type":"ContainerStarted","Data":"7fac3fbd8fce1fc2bd3d39b8550e854eb7b87d44837369246a1bebd15d11fce0"} Nov 03 21:06:23.003604 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:23.003552 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" podStartSLOduration=2.003538718 podStartE2EDuration="2.003538718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:23.001189266 +0000 UTC m=+1132.161261540" watchObservedRunningTime="2026-11-03 21:06:23.003538718 +0000 UTC m=+1132.163610991" Nov 03 21:06:23.104179 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:23.104144 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:26.723647 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:26.723592 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:27.366551 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:27.366517 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:31.442629 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:31.442530 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:32.120335 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:32.120301 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:32.155289 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:32.155252 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:32.984600 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:32.984569 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 21:06:36.100485 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:36.100449 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:39.599011 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:39.598979 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-829xs-9cctk"] Nov 03 21:06:39.602803 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:39.602783 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-829xs-9cctk" Nov 03 21:06:39.658102 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:39.658070 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-829xs-9cctk"] Nov 03 21:06:39.754328 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:39.754287 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c3efa721-99bb-4580-bc52-aee1d344fee9-archives-path\") pod \"periodic-gathering-829xs-9cctk\" (UID: \"c3efa721-99bb-4580-bc52-aee1d344fee9\") " pod="openshift-insights/periodic-gathering-829xs-9cctk" Nov 03 21:06:39.754328 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:39.754330 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hpl9\" (UniqueName: \"kubernetes.io/projected/c3efa721-99bb-4580-bc52-aee1d344fee9-kube-api-access-9hpl9\") pod \"periodic-gathering-829xs-9cctk\" (UID: \"c3efa721-99bb-4580-bc52-aee1d344fee9\") " pod="openshift-insights/periodic-gathering-829xs-9cctk" Nov 03 21:06:39.754626 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:39.754602 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3efa721-99bb-4580-bc52-aee1d344fee9-trusted-ca-bundle\") pod \"periodic-gathering-829xs-9cctk\" (UID: \"c3efa721-99bb-4580-bc52-aee1d344fee9\") " pod="openshift-insights/periodic-gathering-829xs-9cctk" Nov 03 21:06:39.754690 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:39.754662 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3efa721-99bb-4580-bc52-aee1d344fee9-service-ca-bundle\") pod \"periodic-gathering-829xs-9cctk\" (UID: \"c3efa721-99bb-4580-bc52-aee1d344fee9\") " pod="openshift-insights/periodic-gathering-829xs-9cctk" Nov 03 21:06:39.856095 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:39.856006 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3efa721-99bb-4580-bc52-aee1d344fee9-trusted-ca-bundle\") pod \"periodic-gathering-829xs-9cctk\" (UID: \"c3efa721-99bb-4580-bc52-aee1d344fee9\") " pod="openshift-insights/periodic-gathering-829xs-9cctk" Nov 03 21:06:39.856095 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:39.856058 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3efa721-99bb-4580-bc52-aee1d344fee9-service-ca-bundle\") pod \"periodic-gathering-829xs-9cctk\" (UID: \"c3efa721-99bb-4580-bc52-aee1d344fee9\") " pod="openshift-insights/periodic-gathering-829xs-9cctk" Nov 03 21:06:39.856322 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:39.856246 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c3efa721-99bb-4580-bc52-aee1d344fee9-archives-path\") pod \"periodic-gathering-829xs-9cctk\" (UID: \"c3efa721-99bb-4580-bc52-aee1d344fee9\") " pod="openshift-insights/periodic-gathering-829xs-9cctk" Nov 03 21:06:39.856322 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:39.856292 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hpl9\" (UniqueName: \"kubernetes.io/projected/c3efa721-99bb-4580-bc52-aee1d344fee9-kube-api-access-9hpl9\") pod \"periodic-gathering-829xs-9cctk\" (UID: \"c3efa721-99bb-4580-bc52-aee1d344fee9\") " pod="openshift-insights/periodic-gathering-829xs-9cctk" Nov 03 21:06:39.856710 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:39.856688 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c3efa721-99bb-4580-bc52-aee1d344fee9-archives-path\") pod \"periodic-gathering-829xs-9cctk\" (UID: \"c3efa721-99bb-4580-bc52-aee1d344fee9\") " pod="openshift-insights/periodic-gathering-829xs-9cctk" Nov 03 21:06:39.856791 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:39.856773 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3efa721-99bb-4580-bc52-aee1d344fee9-trusted-ca-bundle\") pod \"periodic-gathering-829xs-9cctk\" (UID: \"c3efa721-99bb-4580-bc52-aee1d344fee9\") " pod="openshift-insights/periodic-gathering-829xs-9cctk" Nov 03 21:06:39.856827 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:39.856781 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3efa721-99bb-4580-bc52-aee1d344fee9-service-ca-bundle\") pod \"periodic-gathering-829xs-9cctk\" (UID: \"c3efa721-99bb-4580-bc52-aee1d344fee9\") " pod="openshift-insights/periodic-gathering-829xs-9cctk" Nov 03 21:06:39.915196 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:39.915168 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hpl9\" (UniqueName: \"kubernetes.io/projected/c3efa721-99bb-4580-bc52-aee1d344fee9-kube-api-access-9hpl9\") pod \"periodic-gathering-829xs-9cctk\" (UID: \"c3efa721-99bb-4580-bc52-aee1d344fee9\") " pod="openshift-insights/periodic-gathering-829xs-9cctk" Nov 03 21:06:40.215737 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:40.215697 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-829xs-9cctk" Nov 03 21:06:40.590611 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:40.590525 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-829xs-9cctk"] Nov 03 21:06:40.596384 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:06:40.596359 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3efa721_99bb_4580_bc52_aee1d344fee9.slice/crio-2ea5f7e4dc7909c4852666fdbe3dac264a25504e73560459f68f5f356778208a WatchSource:0}: Error finding container 2ea5f7e4dc7909c4852666fdbe3dac264a25504e73560459f68f5f356778208a: Status 404 returned error can't find the container with id 2ea5f7e4dc7909c4852666fdbe3dac264a25504e73560459f68f5f356778208a Nov 03 21:06:40.618144 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:40.618116 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:41.017391 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:41.017347 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-829xs-9cctk" event={"ID":"c3efa721-99bb-4580-bc52-aee1d344fee9","Type":"ContainerStarted","Data":"81ca2d74316893f65ab6eb8ae7ba4d6a376beea29bf2548da54fe9a385e8b6ef"} Nov 03 21:06:41.017567 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:41.017404 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-829xs-9cctk" event={"ID":"c3efa721-99bb-4580-bc52-aee1d344fee9","Type":"ContainerStarted","Data":"2ea5f7e4dc7909c4852666fdbe3dac264a25504e73560459f68f5f356778208a"} Nov 03 21:06:41.127845 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:41.127783 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-829xs-9cctk" podStartSLOduration=2.127770461 podStartE2EDuration="2.127770461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:41.12664277 +0000 UTC m=+1150.286715043" watchObservedRunningTime="2026-11-03 21:06:41.127770461 +0000 UTC m=+1150.287842732" Nov 03 21:06:42.527874 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:06:42.527837 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:06:45.109558 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:45.109519 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:49.718550 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:49.718519 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:51.228272 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.228235 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9"] Nov 03 21:06:51.232292 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.232271 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.275170 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.275128 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.275366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.275174 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.275366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.275282 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn7zm\" (UniqueName: \"kubernetes.io/projected/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-kube-api-access-jn7zm\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.275366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.275341 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.275562 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.275534 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.275609 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.275587 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.275664 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.275646 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.275722 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.275698 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.275788 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.275765 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.282174 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.282146 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a41ad0-openshift-default-dockercfg-dhkbv\"" Nov 03 21:06:51.333861 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.333823 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9"] Nov 03 21:06:51.376486 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.376453 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.376685 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.376496 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.376685 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.376522 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.376685 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.376561 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.376875 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.376705 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.376875 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.376755 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.376875 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.376783 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.376875 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.376840 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn7zm\" (UniqueName: \"kubernetes.io/projected/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-kube-api-access-jn7zm\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.377133 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.376930 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.377989 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.377937 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.378107 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.378032 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.379525 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.379367 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.379616 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.379565 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.379616 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.377215 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.379980 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.379955 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.381132 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.381113 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.435149 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.435120 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.435296 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.435283 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn7zm\" (UniqueName: \"kubernetes.io/projected/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-kube-api-access-jn7zm\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.550140 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.550060 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:51.979578 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.979540 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9"] Nov 03 21:06:51.985565 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:06:51.985538 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8b4ff4_9263_4e71_9e1e_c0bbd1a30ee0.slice/crio-47ca428e051b33e79b93b4053058701d6e50ae17a1d10d896ca2a7352547099c WatchSource:0}: Error finding container 47ca428e051b33e79b93b4053058701d6e50ae17a1d10d896ca2a7352547099c: Status 404 returned error can't find the container with id 47ca428e051b33e79b93b4053058701d6e50ae17a1d10d896ca2a7352547099c Nov 03 21:06:51.987864 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.987831 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:06:51.988006 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.987947 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:06:51.988006 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:51.987995 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:06:52.084807 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:52.084773 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" event={"ID":"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0","Type":"ContainerStarted","Data":"71bc0214c392dc67537c95b57804834f25de33c8c6d3761482b4ed444b279806"} Nov 03 21:06:52.084978 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:52.084826 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" event={"ID":"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0","Type":"ContainerStarted","Data":"47ca428e051b33e79b93b4053058701d6e50ae17a1d10d896ca2a7352547099c"} Nov 03 21:06:52.201499 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:52.201445 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" podStartSLOduration=1.201431299 podStartE2EDuration="1.201431299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:52.200945884 +0000 UTC m=+1161.361018155" watchObservedRunningTime="2026-11-03 21:06:52.201431299 +0000 UTC m=+1161.361503570" Nov 03 21:06:52.550745 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:52.550711 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:53.551951 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:53.551888 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 21:06:53.552429 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:53.551985 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" podUID="8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 21:06:54.257765 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:54.257729 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:54.550723 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:54.550640 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 21:06:54.550723 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:54.550711 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" podUID="8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 21:06:55.551200 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:55.551162 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 21:06:55.551621 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:55.551232 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" podUID="8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 21:06:56.081617 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:56.081578 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:56.550529 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:56.550492 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 21:06:56.550712 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:56.550546 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" podUID="8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 21:06:57.551003 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:57.550963 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 21:06:57.551419 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:57.551017 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" podUID="8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 21:06:58.550844 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:58.550807 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 21:06:58.551046 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:58.550864 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" podUID="8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 21:06:58.791710 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:58.791657 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:59.554751 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:59.554718 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:59.555221 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:59.555185 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:06:59.555873 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:06:59.555828 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:07:01.975713 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:01.975669 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:02.162623 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:02.162588 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:03.302050 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:03.302019 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:04.102698 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:04.102637 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:05.942784 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:05.942746 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:07.920116 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:07.920077 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:12.515754 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:12.515693 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:12.528047 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:07:12.528011 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:07:17.101743 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:17.101700 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:20.023841 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:20.023803 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f"] Nov 03 21:07:20.105016 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:20.104977 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:21.569653 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:21.569618 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:26.102119 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:26.102083 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:27.700272 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:27.700240 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:30.715569 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:30.715535 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:31.086751 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:31.086671 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:07:31.092583 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:31.092553 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:07:35.046786 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.046738 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" podUID="7c19e318-7e17-4a5c-8fa5-eced96a4f0fc" containerName="main" containerID="cri-o://15203fd312415fca8ac1dcab5a2e24c1ceb2df4839061f4ad294011e235787f3" gracePeriod=45 Nov 03 21:07:35.244749 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.244714 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw"] Nov 03 21:07:35.248984 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.248958 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.311968 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.311869 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:07:35.313465 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.313437 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:35.326094 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.326062 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw"] Nov 03 21:07:35.366531 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.366463 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2b9bb1-4418-46a4-941d-def176133161-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.367343 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.366575 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.367343 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.366624 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5t6q\" (UniqueName: \"kubernetes.io/projected/3c2b9bb1-4418-46a4-941d-def176133161-kube-api-access-s5t6q\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.367343 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.366692 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.367343 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.366726 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.367343 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.366796 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2b9bb1-4418-46a4-941d-def176133161-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.367343 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.366826 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.367343 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.366849 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2b9bb1-4418-46a4-941d-def176133161-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.367343 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.366932 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.435808 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.435774 4963 generic.go:356] "Generic (PLEG): container finished" podID="7c19e318-7e17-4a5c-8fa5-eced96a4f0fc" containerID="15203fd312415fca8ac1dcab5a2e24c1ceb2df4839061f4ad294011e235787f3" exitCode=0 Nov 03 21:07:35.436013 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.435825 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" Nov 03 21:07:35.436013 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.435872 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" event={"ID":"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc","Type":"ContainerDied","Data":"15203fd312415fca8ac1dcab5a2e24c1ceb2df4839061f4ad294011e235787f3"} Nov 03 21:07:35.436013 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.435950 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" event={"ID":"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc","Type":"ContainerDied","Data":"dafce474f073a797b6986a3715fc43a4ce1b8db9398dc827876a2beea8bf6f58"} Nov 03 21:07:35.436013 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.435970 4963 scope.go:122] "RemoveContainer" containerID="15203fd312415fca8ac1dcab5a2e24c1ceb2df4839061f4ad294011e235787f3" Nov 03 21:07:35.445610 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.445587 4963 scope.go:122] "RemoveContainer" containerID="15203fd312415fca8ac1dcab5a2e24c1ceb2df4839061f4ad294011e235787f3" Nov 03 21:07:35.446083 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:07:35.446052 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15203fd312415fca8ac1dcab5a2e24c1ceb2df4839061f4ad294011e235787f3\": container with ID starting with 15203fd312415fca8ac1dcab5a2e24c1ceb2df4839061f4ad294011e235787f3 not found: ID does not exist" containerID="15203fd312415fca8ac1dcab5a2e24c1ceb2df4839061f4ad294011e235787f3" Nov 03 21:07:35.446189 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.446096 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15203fd312415fca8ac1dcab5a2e24c1ceb2df4839061f4ad294011e235787f3"} err="failed to get container status \"15203fd312415fca8ac1dcab5a2e24c1ceb2df4839061f4ad294011e235787f3\": rpc error: code = NotFound desc = could not find container \"15203fd312415fca8ac1dcab5a2e24c1ceb2df4839061f4ad294011e235787f3\": container with ID starting with 15203fd312415fca8ac1dcab5a2e24c1ceb2df4839061f4ad294011e235787f3 not found: ID does not exist" Nov 03 21:07:35.467655 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.467624 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-tls-certs\") pod \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " Nov 03 21:07:35.467833 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.467725 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-model-cache\") pod \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " Nov 03 21:07:35.467833 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.467757 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-tmp-dir\") pod \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " Nov 03 21:07:35.467998 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.467848 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-dshm\") pod \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " Nov 03 21:07:35.467998 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.467878 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-home\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-home\") pod \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " Nov 03 21:07:35.467998 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.467952 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-kube-api-access-pmh97\" (UniqueName: \"kubernetes.io/projected/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-kube-api-access-pmh97\") pod \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\" (UID: \"7c19e318-7e17-4a5c-8fa5-eced96a4f0fc\") " Nov 03 21:07:35.468153 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.468084 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-model-cache" pod "7c19e318-7e17-4a5c-8fa5-eced96a4f0fc" (UID: "7c19e318-7e17-4a5c-8fa5-eced96a4f0fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:35.468198 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.468157 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-tmp-dir" pod "7c19e318-7e17-4a5c-8fa5-eced96a4f0fc" (UID: "7c19e318-7e17-4a5c-8fa5-eced96a4f0fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:35.468247 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.468164 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.468310 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.468244 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-home" pod "7c19e318-7e17-4a5c-8fa5-eced96a4f0fc" (UID: "7c19e318-7e17-4a5c-8fa5-eced96a4f0fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:35.468360 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.468334 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2b9bb1-4418-46a4-941d-def176133161-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.468438 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.468414 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.468499 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.468453 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5t6q\" (UniqueName: \"kubernetes.io/projected/3c2b9bb1-4418-46a4-941d-def176133161-kube-api-access-s5t6q\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.468554 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.468501 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.468554 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.468503 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.468554 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.468525 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.468701 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.468596 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2b9bb1-4418-46a4-941d-def176133161-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.468701 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.468634 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.468701 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.468662 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2b9bb1-4418-46a4-941d-def176133161-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.468948 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.468742 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:35.468948 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.468747 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.468948 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.468759 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:35.468948 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.468773 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:35.469174 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.469124 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2b9bb1-4418-46a4-941d-def176133161-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.469339 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.469310 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.469496 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.469461 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.470364 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.470340 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-tls-certs" pod "7c19e318-7e17-4a5c-8fa5-eced96a4f0fc" (UID: "7c19e318-7e17-4a5c-8fa5-eced96a4f0fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:07:35.470461 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.470412 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-kube-api-access-pmh97" pod "7c19e318-7e17-4a5c-8fa5-eced96a4f0fc" (UID: "7c19e318-7e17-4a5c-8fa5-eced96a4f0fc"). InnerVolumeSpecName "kube-api-access-pmh97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:35.470505 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.470457 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-dshm" pod "7c19e318-7e17-4a5c-8fa5-eced96a4f0fc" (UID: "7c19e318-7e17-4a5c-8fa5-eced96a4f0fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:35.471263 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.471242 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.471263 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.471254 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2b9bb1-4418-46a4-941d-def176133161-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.527362 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.527327 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2b9bb1-4418-46a4-941d-def176133161-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.527535 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.527365 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5t6q\" (UniqueName: \"kubernetes.io/projected/3c2b9bb1-4418-46a4-941d-def176133161-kube-api-access-s5t6q\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.566261 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.566166 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:35.569768 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.569738 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmh97\" (UniqueName: \"kubernetes.io/projected/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-kube-api-access-pmh97\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:35.569768 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.569769 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:35.569984 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.569781 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:35.856024 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.855936 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f"] Nov 03 21:07:35.862167 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:35.862135 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f"] Nov 03 21:07:36.024286 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:36.024248 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw"] Nov 03 21:07:36.029787 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:07:36.029752 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2b9bb1_4418_46a4_941d_def176133161.slice/crio-a694089071deadff135697f4c7d4d07e8683dc6cb2ec49a4d8050dfb3d79bf82 WatchSource:0}: Error finding container a694089071deadff135697f4c7d4d07e8683dc6cb2ec49a4d8050dfb3d79bf82: Status 404 returned error can't find the container with id a694089071deadff135697f4c7d4d07e8683dc6cb2ec49a4d8050dfb3d79bf82 Nov 03 21:07:36.032284 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:36.032242 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:07:36.032368 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:36.032309 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:07:36.032368 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:36.032347 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:07:36.230816 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:36.230773 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-a-3ba3a28c-kserve-766957f594-wb89f" podUID="7c19e318-7e17-4a5c-8fa5-eced96a4f0fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:36.445626 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:36.445578 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" event={"ID":"3c2b9bb1-4418-46a4-941d-def176133161","Type":"ContainerStarted","Data":"de89f7eecfeeac97967c1a7164d13e58bd95c5c9b08bb8481110316153107962"} Nov 03 21:07:36.445828 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:36.445637 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" event={"ID":"3c2b9bb1-4418-46a4-941d-def176133161","Type":"ContainerStarted","Data":"a694089071deadff135697f4c7d4d07e8683dc6cb2ec49a4d8050dfb3d79bf82"} Nov 03 21:07:36.564361 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:36.564282 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" podStartSLOduration=1.5642591879999999 podStartE2EDuration="1.564259188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:36.559574294 +0000 UTC m=+1205.719646566" watchObservedRunningTime="2026-11-03 21:07:36.564259188 +0000 UTC m=+1205.724331485" Nov 03 21:07:36.567010 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:36.566983 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:36.967624 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:36.967591 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c19e318-7e17-4a5c-8fa5-eced96a4f0fc" path="/var/lib/kubelet/pods/7c19e318-7e17-4a5c-8fa5-eced96a4f0fc/volumes" Nov 03 21:07:37.569038 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:37.569005 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 21:07:37.569446 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:37.569058 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" podUID="3c2b9bb1-4418-46a4-941d-def176133161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 21:07:38.567058 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:38.567023 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 21:07:38.567240 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:38.567077 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" podUID="3c2b9bb1-4418-46a4-941d-def176133161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 21:07:39.566684 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:39.566649 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 21:07:39.567215 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:39.566705 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" podUID="3c2b9bb1-4418-46a4-941d-def176133161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 21:07:39.959156 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:39.959129 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:40.566876 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:40.566832 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 21:07:40.567305 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:40.566939 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" podUID="3c2b9bb1-4418-46a4-941d-def176133161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 21:07:41.567137 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:41.567104 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 21:07:41.567553 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:41.567156 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" podUID="3c2b9bb1-4418-46a4-941d-def176133161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 21:07:42.528349 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:07:42.528308 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:07:42.566971 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:42.566928 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 21:07:42.567174 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:42.566985 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" podUID="3c2b9bb1-4418-46a4-941d-def176133161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 21:07:43.571475 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:43.571443 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:43.571992 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:43.571826 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:43.572608 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:43.572587 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:07:44.528714 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:44.528665 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:49.122734 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:49.122695 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:51.243680 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.243640 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl"] Nov 03 21:07:51.244368 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.244347 4963 cpu_manager.go:414] "containerMap: removing container" podUID="7c19e318-7e17-4a5c-8fa5-eced96a4f0fc" containerName="main" Nov 03 21:07:51.244466 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.244370 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c19e318-7e17-4a5c-8fa5-eced96a4f0fc" containerName="main" Nov 03 21:07:51.244540 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.244527 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c19e318-7e17-4a5c-8fa5-eced96a4f0fc" containerName="main" Nov 03 21:07:51.248125 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.248104 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.373402 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.373356 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.373614 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.373411 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccxrq\" (UniqueName: \"kubernetes.io/projected/955fcff3-0d65-4115-916a-18c1506a3aba-kube-api-access-ccxrq\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.373614 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.373496 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.373614 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.373558 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.373614 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.373586 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955fcff3-0d65-4115-916a-18c1506a3aba-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.373794 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.373681 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.373794 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.373722 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/955fcff3-0d65-4115-916a-18c1506a3aba-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.373882 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.373808 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955fcff3-0d65-4115-916a-18c1506a3aba-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.373882 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.373857 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.448703 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.448663 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl"] Nov 03 21:07:51.474809 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.474769 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.474809 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.474809 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccxrq\" (UniqueName: \"kubernetes.io/projected/955fcff3-0d65-4115-916a-18c1506a3aba-kube-api-access-ccxrq\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.475109 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.474830 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.475109 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.474851 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.475109 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.474867 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955fcff3-0d65-4115-916a-18c1506a3aba-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.475109 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.475054 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.476024 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.475281 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.476024 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.475388 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.476024 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.475507 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/955fcff3-0d65-4115-916a-18c1506a3aba-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.476024 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.475521 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.476024 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.475610 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955fcff3-0d65-4115-916a-18c1506a3aba-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.476024 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.475670 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.476336 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.476034 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.476432 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.476410 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955fcff3-0d65-4115-916a-18c1506a3aba-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.477778 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.477749 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955fcff3-0d65-4115-916a-18c1506a3aba-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.478175 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.478150 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.533850 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.533771 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/955fcff3-0d65-4115-916a-18c1506a3aba-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.534011 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.533895 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccxrq\" (UniqueName: \"kubernetes.io/projected/955fcff3-0d65-4115-916a-18c1506a3aba-kube-api-access-ccxrq\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.562103 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.562076 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:51.995918 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:51.995883 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl"] Nov 03 21:07:52.001642 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:07:52.001606 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955fcff3_0d65_4115_916a_18c1506a3aba.slice/crio-cb6f2832a47aa395ebb36e7a48a0a4a462631dac5073be7fa2fd93c6816164fc WatchSource:0}: Error finding container cb6f2832a47aa395ebb36e7a48a0a4a462631dac5073be7fa2fd93c6816164fc: Status 404 returned error can't find the container with id cb6f2832a47aa395ebb36e7a48a0a4a462631dac5073be7fa2fd93c6816164fc Nov 03 21:07:52.004159 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:52.004126 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:07:52.004243 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:52.004196 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:07:52.004243 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:52.004226 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:07:52.550851 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:52.550811 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" event={"ID":"955fcff3-0d65-4115-916a-18c1506a3aba","Type":"ContainerStarted","Data":"6c53948cee8c0689fdbbde22ff64af1a887ef0d2606b959f8bb1fd86446339e9"} Nov 03 21:07:52.551376 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:52.550883 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" event={"ID":"955fcff3-0d65-4115-916a-18c1506a3aba","Type":"ContainerStarted","Data":"cb6f2832a47aa395ebb36e7a48a0a4a462631dac5073be7fa2fd93c6816164fc"} Nov 03 21:07:52.562197 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:52.562163 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:52.710929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:52.709775 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" podStartSLOduration=1.709757321 podStartE2EDuration="1.709757321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:52.672780616 +0000 UTC m=+1221.832852888" watchObservedRunningTime="2026-11-03 21:07:52.709757321 +0000 UTC m=+1221.869829593" Nov 03 21:07:53.564186 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:53.564155 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:07:53.564599 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:53.564205 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" podUID="955fcff3-0d65-4115-916a-18c1506a3aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 21:07:53.731447 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:53.731414 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:54.562760 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:54.562730 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:07:54.562958 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:54.562778 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" podUID="955fcff3-0d65-4115-916a-18c1506a3aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 21:07:55.562873 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:55.562834 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:07:55.563427 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:55.562891 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" podUID="955fcff3-0d65-4115-916a-18c1506a3aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 21:07:56.562958 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:56.562879 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:07:56.563515 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:56.562995 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" podUID="955fcff3-0d65-4115-916a-18c1506a3aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 21:07:57.563462 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:57.563429 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:07:57.563853 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:57.563480 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" podUID="955fcff3-0d65-4115-916a-18c1506a3aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 21:07:58.334556 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:58.334520 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:58.562441 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:58.562408 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 21:07:58.562613 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:58.562473 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" podUID="955fcff3-0d65-4115-916a-18c1506a3aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 21:07:59.566625 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:59.566594 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:59.567132 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:59.566952 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:07:59.568017 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:07:59.567996 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:08:02.971090 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:02.971051 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:05.485674 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.485629 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd"] Nov 03 21:08:05.489986 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.489963 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.573754 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.573716 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.573997 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.573781 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda7b470-9598-458c-a270-4b0789924f39-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.573997 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.573820 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.573997 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.573867 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.573997 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.573876 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd"] Nov 03 21:08:05.574224 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.574061 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda7b470-9598-458c-a270-4b0789924f39-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.574224 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.574095 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda7b470-9598-458c-a270-4b0789924f39-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.574224 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.574127 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.574224 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.574147 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.574224 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.574184 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp5rs\" (UniqueName: \"kubernetes.io/projected/eda7b470-9598-458c-a270-4b0789924f39-kube-api-access-bp5rs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.674820 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.674786 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda7b470-9598-458c-a270-4b0789924f39-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.674820 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.674829 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda7b470-9598-458c-a270-4b0789924f39-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.675119 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.674851 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.675119 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.674871 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.675119 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.674933 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp5rs\" (UniqueName: \"kubernetes.io/projected/eda7b470-9598-458c-a270-4b0789924f39-kube-api-access-bp5rs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.675119 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.674985 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.675119 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.675015 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda7b470-9598-458c-a270-4b0789924f39-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.675119 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.675056 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.675119 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.675114 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.675508 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.675410 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.675508 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.675477 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.675621 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.675573 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.675621 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.675592 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.675724 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.675705 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda7b470-9598-458c-a270-4b0789924f39-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.677252 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.677228 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.677567 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.677543 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda7b470-9598-458c-a270-4b0789924f39-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.735080 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.735050 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda7b470-9598-458c-a270-4b0789924f39-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.735277 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.735184 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp5rs\" (UniqueName: \"kubernetes.io/projected/eda7b470-9598-458c-a270-4b0789924f39-kube-api-access-bp5rs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:05.804274 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:05.804187 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:06.231369 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:06.231336 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd"] Nov 03 21:08:06.236693 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:08:06.236663 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda7b470_9598_458c_a270_4b0789924f39.slice/crio-43564b23f0fc34ded58125f37befb6ec8f47f616344d82a44a43aa10383f923a WatchSource:0}: Error finding container 43564b23f0fc34ded58125f37befb6ec8f47f616344d82a44a43aa10383f923a: Status 404 returned error can't find the container with id 43564b23f0fc34ded58125f37befb6ec8f47f616344d82a44a43aa10383f923a Nov 03 21:08:06.239236 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:06.239202 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:08:06.239330 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:06.239271 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:08:06.239330 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:06.239298 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:08:06.657855 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:06.657766 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" event={"ID":"eda7b470-9598-458c-a270-4b0789924f39","Type":"ContainerStarted","Data":"14f2b039483f8e137cd328724290d890cbd3bfbd8a5b1071872221c50b989b90"} Nov 03 21:08:06.657855 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:06.657831 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" event={"ID":"eda7b470-9598-458c-a270-4b0789924f39","Type":"ContainerStarted","Data":"43564b23f0fc34ded58125f37befb6ec8f47f616344d82a44a43aa10383f923a"} Nov 03 21:08:06.775673 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:06.775617 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" podStartSLOduration=1.775602293 podStartE2EDuration="1.775602293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:06.774959758 +0000 UTC m=+1235.935032032" watchObservedRunningTime="2026-11-03 21:08:06.775602293 +0000 UTC m=+1235.935674565" Nov 03 21:08:06.805025 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:06.804988 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:07.630628 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:07.630597 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:07.809925 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:07.809803 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:08:07.810433 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:07.809893 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" podUID="eda7b470-9598-458c-a270-4b0789924f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 21:08:08.806495 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:08.806460 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:08:08.806717 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:08.806513 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" podUID="eda7b470-9598-458c-a270-4b0789924f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 21:08:09.806408 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:09.806369 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:08:09.806894 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:09.806436 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" podUID="eda7b470-9598-458c-a270-4b0789924f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 21:08:10.806385 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:10.806340 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:08:10.806712 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:10.806409 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" podUID="eda7b470-9598-458c-a270-4b0789924f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 21:08:11.806334 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:11.806297 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:08:11.806523 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:11.806354 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" podUID="eda7b470-9598-458c-a270-4b0789924f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 21:08:12.303284 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:12.303243 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:12.535936 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:08:12.535884 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:08:12.805502 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:12.805463 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 21:08:12.805685 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:12.805524 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" podUID="eda7b470-9598-458c-a270-4b0789924f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 21:08:12.974260 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:12.974226 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:13.809943 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:13.809889 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:13.810391 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:13.810192 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:13.811048 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:13.811027 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 21:08:16.301713 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.301676 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q"] Nov 03 21:08:16.328655 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.328618 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.397072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.397032 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-099a5e\"/\"distinct-w2-099a5e-kserve-self-signed-certs\"" Nov 03 21:08:16.454385 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.454348 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:16.455506 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.455477 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-model-cache\") pod \"distinct-w2-099a5e-kserve-6b5759886d-c7w4q\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.510926 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.510046 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-tls-certs\") pod \"distinct-w2-099a5e-kserve-6b5759886d-c7w4q\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.510926 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.510117 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-tmp-dir\") pod \"distinct-w2-099a5e-kserve-6b5759886d-c7w4q\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.510926 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.510203 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-dshm\") pod \"distinct-w2-099a5e-kserve-6b5759886d-c7w4q\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.510926 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.510288 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-home\") pod \"distinct-w2-099a5e-kserve-6b5759886d-c7w4q\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.510926 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.510384 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jw9x\" (UniqueName: \"kubernetes.io/projected/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-kube-api-access-4jw9x\") pod \"distinct-w2-099a5e-kserve-6b5759886d-c7w4q\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.535986 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.535953 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q"] Nov 03 21:08:16.640069 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.613174 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-dshm\") pod \"distinct-w2-099a5e-kserve-6b5759886d-c7w4q\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.675435 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.675398 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-home\") pod \"distinct-w2-099a5e-kserve-6b5759886d-c7w4q\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.676210 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.676184 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jw9x\" (UniqueName: \"kubernetes.io/projected/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-kube-api-access-4jw9x\") pod \"distinct-w2-099a5e-kserve-6b5759886d-c7w4q\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.678714 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.678687 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-model-cache\") pod \"distinct-w2-099a5e-kserve-6b5759886d-c7w4q\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.679285 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.679265 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-tls-certs\") pod \"distinct-w2-099a5e-kserve-6b5759886d-c7w4q\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.679950 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.679930 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-tmp-dir\") pod \"distinct-w2-099a5e-kserve-6b5759886d-c7w4q\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.680579 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.679193 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-model-cache\") pod \"distinct-w2-099a5e-kserve-6b5759886d-c7w4q\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.680721 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.680427 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-tmp-dir\") pod \"distinct-w2-099a5e-kserve-6b5759886d-c7w4q\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.680834 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.676077 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-home\") pod \"distinct-w2-099a5e-kserve-6b5759886d-c7w4q\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.771051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.770612 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-tls-certs\") pod \"distinct-w2-099a5e-kserve-6b5759886d-c7w4q\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.771945 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.771895 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-dshm\") pod \"distinct-w2-099a5e-kserve-6b5759886d-c7w4q\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.830928 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.823966 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jw9x\" (UniqueName: \"kubernetes.io/projected/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-kube-api-access-4jw9x\") pod \"distinct-w2-099a5e-kserve-6b5759886d-c7w4q\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:16.864952 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.864915 4963 generic.go:356] "Generic (PLEG): container finished" podID="c3efa721-99bb-4580-bc52-aee1d344fee9" containerID="81ca2d74316893f65ab6eb8ae7ba4d6a376beea29bf2548da54fe9a385e8b6ef" exitCode=0 Nov 03 21:08:16.865224 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:16.865201 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-829xs-9cctk" event={"ID":"c3efa721-99bb-4580-bc52-aee1d344fee9","Type":"ContainerDied","Data":"81ca2d74316893f65ab6eb8ae7ba4d6a376beea29bf2548da54fe9a385e8b6ef"} Nov 03 21:08:17.036930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:17.033087 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:17.651989 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:17.651882 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q"] Nov 03 21:08:17.658560 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:08:17.658476 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4524d0a6_727b_47a9_b3b9_e83e5cf3a52d.slice/crio-4e76a9a8c0a804303883ef897fb8c540b4748c439367ffb43668ca0c4cdbe243 WatchSource:0}: Error finding container 4e76a9a8c0a804303883ef897fb8c540b4748c439367ffb43668ca0c4cdbe243: Status 404 returned error can't find the container with id 4e76a9a8c0a804303883ef897fb8c540b4748c439367ffb43668ca0c4cdbe243 Nov 03 21:08:17.768604 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:17.768485 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:17.872765 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:17.872730 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" event={"ID":"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d","Type":"ContainerStarted","Data":"76861975ac0597c193c769913dfb0fec6dbf2f0e3fbfa1428d069c722139f015"} Nov 03 21:08:17.872976 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:17.872774 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" event={"ID":"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d","Type":"ContainerStarted","Data":"4e76a9a8c0a804303883ef897fb8c540b4748c439367ffb43668ca0c4cdbe243"} Nov 03 21:08:17.985900 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:17.985846 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" podStartSLOduration=1.985832936 podStartE2EDuration="1.985832936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:17.9844775 +0000 UTC m=+1247.144549772" watchObservedRunningTime="2026-11-03 21:08:17.985832936 +0000 UTC m=+1247.145905207" Nov 03 21:08:18.032489 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:18.032461 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-829xs-9cctk" Nov 03 21:08:18.035229 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:18.035204 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:18.172834 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:18.172798 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3efa721-99bb-4580-bc52-aee1d344fee9-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c3efa721-99bb-4580-bc52-aee1d344fee9-archives-path\") pod \"c3efa721-99bb-4580-bc52-aee1d344fee9\" (UID: \"c3efa721-99bb-4580-bc52-aee1d344fee9\") " Nov 03 21:08:18.173027 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:18.172948 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3efa721-99bb-4580-bc52-aee1d344fee9-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3efa721-99bb-4580-bc52-aee1d344fee9-service-ca-bundle\") pod \"c3efa721-99bb-4580-bc52-aee1d344fee9\" (UID: \"c3efa721-99bb-4580-bc52-aee1d344fee9\") " Nov 03 21:08:18.173080 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:18.173034 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3efa721-99bb-4580-bc52-aee1d344fee9-kube-api-access-9hpl9\" (UniqueName: \"kubernetes.io/projected/c3efa721-99bb-4580-bc52-aee1d344fee9-kube-api-access-9hpl9\") pod \"c3efa721-99bb-4580-bc52-aee1d344fee9\" (UID: \"c3efa721-99bb-4580-bc52-aee1d344fee9\") " Nov 03 21:08:18.173166 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:18.173150 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3efa721-99bb-4580-bc52-aee1d344fee9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3efa721-99bb-4580-bc52-aee1d344fee9-trusted-ca-bundle\") pod \"c3efa721-99bb-4580-bc52-aee1d344fee9\" (UID: \"c3efa721-99bb-4580-bc52-aee1d344fee9\") " Nov 03 21:08:18.173229 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:18.173208 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3efa721-99bb-4580-bc52-aee1d344fee9-archives-path" pod "c3efa721-99bb-4580-bc52-aee1d344fee9" (UID: "c3efa721-99bb-4580-bc52-aee1d344fee9"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:18.173333 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:18.173304 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3efa721-99bb-4580-bc52-aee1d344fee9-service-ca-bundle" pod "c3efa721-99bb-4580-bc52-aee1d344fee9" (UID: "c3efa721-99bb-4580-bc52-aee1d344fee9"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:18.173546 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:18.173525 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3efa721-99bb-4580-bc52-aee1d344fee9-trusted-ca-bundle" pod "c3efa721-99bb-4580-bc52-aee1d344fee9" (UID: "c3efa721-99bb-4580-bc52-aee1d344fee9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:18.173674 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:18.173660 4963 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3efa721-99bb-4580-bc52-aee1d344fee9-service-ca-bundle\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:18.173725 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:18.173677 4963 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3efa721-99bb-4580-bc52-aee1d344fee9-trusted-ca-bundle\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:18.173725 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:18.173688 4963 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c3efa721-99bb-4580-bc52-aee1d344fee9-archives-path\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:18.175358 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:18.175335 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3efa721-99bb-4580-bc52-aee1d344fee9-kube-api-access-9hpl9" pod "c3efa721-99bb-4580-bc52-aee1d344fee9" (UID: "c3efa721-99bb-4580-bc52-aee1d344fee9"). InnerVolumeSpecName "kube-api-access-9hpl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:18.274596 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:18.274558 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hpl9\" (UniqueName: \"kubernetes.io/projected/c3efa721-99bb-4580-bc52-aee1d344fee9-kube-api-access-9hpl9\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:18.880165 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:18.880131 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-829xs-9cctk" Nov 03 21:08:18.880165 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:18.880176 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-829xs-9cctk" event={"ID":"c3efa721-99bb-4580-bc52-aee1d344fee9","Type":"ContainerDied","Data":"2ea5f7e4dc7909c4852666fdbe3dac264a25504e73560459f68f5f356778208a"} Nov 03 21:08:18.880729 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:18.880216 4963 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ea5f7e4dc7909c4852666fdbe3dac264a25504e73560459f68f5f356778208a" Nov 03 21:08:19.944122 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:19.944089 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:20.129499 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:20.129460 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:22.223598 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:22.223526 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:26.703577 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:26.703541 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:27.035974 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:27.035847 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:27.050421 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:27.050380 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:27.989222 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:27.989191 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 21:08:28.081278 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:28.081238 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:29.855927 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:29.855065 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:08:29.855927 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:29.855140 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" podUID="eda7b470-9598-458c-a270-4b0789924f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:08:31.714134 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:31.714100 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:35.307822 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:35.307774 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:36.241479 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.241443 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l"] Nov 03 21:08:36.242188 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.242163 4963 cpu_manager.go:414] "containerMap: removing container" podUID="c3efa721-99bb-4580-bc52-aee1d344fee9" containerName="insights-gathering" Nov 03 21:08:36.242188 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.242186 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3efa721-99bb-4580-bc52-aee1d344fee9" containerName="insights-gathering" Nov 03 21:08:36.242350 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.242290 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3efa721-99bb-4580-bc52-aee1d344fee9" containerName="insights-gathering" Nov 03 21:08:36.247458 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.247432 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.267033 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.266989 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:36.347938 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.347892 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l"] Nov 03 21:08:36.362143 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.362111 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.362311 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.362216 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.362311 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.362273 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e68d020-3255-49f5-b986-b2f07c317894-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.362311 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.362296 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kprn\" (UniqueName: \"kubernetes.io/projected/8e68d020-3255-49f5-b986-b2f07c317894-kube-api-access-9kprn\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.362467 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.362443 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e68d020-3255-49f5-b986-b2f07c317894-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.362545 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.362532 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.362589 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.362560 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.362634 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.362617 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.362690 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.362644 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e68d020-3255-49f5-b986-b2f07c317894-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.464341 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.464296 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.464550 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.464346 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.464550 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.464380 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.464550 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.464406 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e68d020-3255-49f5-b986-b2f07c317894-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.464550 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.464457 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.464550 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.464520 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.464815 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.464580 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e68d020-3255-49f5-b986-b2f07c317894-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.464815 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.464614 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kprn\" (UniqueName: \"kubernetes.io/projected/8e68d020-3255-49f5-b986-b2f07c317894-kube-api-access-9kprn\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.464815 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.464680 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e68d020-3255-49f5-b986-b2f07c317894-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.464981 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.464859 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.465293 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.465267 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.465293 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.465283 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e68d020-3255-49f5-b986-b2f07c317894-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.465464 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.465304 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.465464 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.465353 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.466856 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.466834 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.467602 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.467586 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e68d020-3255-49f5-b986-b2f07c317894-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.523741 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.523671 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e68d020-3255-49f5-b986-b2f07c317894-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.523894 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.523817 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kprn\" (UniqueName: \"kubernetes.io/projected/8e68d020-3255-49f5-b986-b2f07c317894-kube-api-access-9kprn\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.563155 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.563120 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:36.991126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:36.991076 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l"] Nov 03 21:08:36.998289 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:08:36.997527 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e68d020_3255_49f5_b986_b2f07c317894.slice/crio-5f6e18790ba924914270a1ccf9f8df2197cd1757d650128f4d142884e2797688 WatchSource:0}: Error finding container 5f6e18790ba924914270a1ccf9f8df2197cd1757d650128f4d142884e2797688: Status 404 returned error can't find the container with id 5f6e18790ba924914270a1ccf9f8df2197cd1757d650128f4d142884e2797688 Nov 03 21:08:37.000778 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:37.000742 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:08:37.000897 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:37.000822 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:08:37.000897 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:37.000852 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:08:37.106262 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:37.106222 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" event={"ID":"8e68d020-3255-49f5-b986-b2f07c317894","Type":"ContainerStarted","Data":"27d12de9ee59adff058c8a103516ff12e10f51322f07cf92ce8510608c6c338d"} Nov 03 21:08:37.106416 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:37.106288 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" event={"ID":"8e68d020-3255-49f5-b986-b2f07c317894","Type":"ContainerStarted","Data":"5f6e18790ba924914270a1ccf9f8df2197cd1757d650128f4d142884e2797688"} Nov 03 21:08:37.225296 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:37.225238 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" podStartSLOduration=1.225224151 podStartE2EDuration="1.225224151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:37.223835821 +0000 UTC m=+1266.383908095" watchObservedRunningTime="2026-11-03 21:08:37.225224151 +0000 UTC m=+1266.385296422" Nov 03 21:08:37.563562 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:37.563523 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:38.565545 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:38.565510 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:08:38.565982 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:38.565567 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" podUID="8e68d020-3255-49f5-b986-b2f07c317894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 21:08:39.564425 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:39.564387 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:08:39.564619 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:39.564441 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" podUID="8e68d020-3255-49f5-b986-b2f07c317894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 21:08:40.564416 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:40.564380 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:08:40.564836 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:40.564433 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" podUID="8e68d020-3255-49f5-b986-b2f07c317894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 21:08:40.842069 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:40.841900 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:41.564365 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:41.564326 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:08:41.564548 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:41.564381 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" podUID="8e68d020-3255-49f5-b986-b2f07c317894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 21:08:42.534872 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:08:42.534834 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:08:42.563605 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:42.563570 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:08:42.563779 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:42.563628 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" podUID="8e68d020-3255-49f5-b986-b2f07c317894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 21:08:43.573929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:43.564359 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:08:43.573929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:43.564421 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" podUID="8e68d020-3255-49f5-b986-b2f07c317894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 21:08:44.566332 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:44.566296 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:08:44.566648 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:44.566615 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" podUID="8e68d020-3255-49f5-b986-b2f07c317894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 21:08:44.981931 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:44.981064 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" podUID="7bdfcd76-3dd3-46fd-a382-0e40e648aa17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:08:45.176930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:45.176047 4963 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" podUID="662bceeb-beae-4a4f-b08a-08706bed2d2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:08:45.209929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:45.205011 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" podUID="1edcbec2-0930-4b49-b14a-2d21568b5318" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:08:45.427184 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:45.427070 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" podUID="fa8baac2-125d-42f6-8449-5fd590bf4b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:08:46.302523 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:46.302491 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:46.524020 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:46.523978 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:46.605181 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:46.605074 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:08:46.605181 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:46.605143 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" podUID="8e68d020-3255-49f5-b986-b2f07c317894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:08:46.975221 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:46.975179 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:46.975563 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:46.975540 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:46.982080 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:46.982042 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:08:47.269111 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.269031 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f"] Nov 03 21:08:47.273105 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.273078 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.323180 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.323142 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-099a5e\"/\"distinct-2-w2-099a5e-kserve-self-signed-certs\"" Nov 03 21:08:47.332297 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.332256 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f"] Nov 03 21:08:47.357466 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.357426 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-tls-certs\") pod \"distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.357661 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.357477 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-home\") pod \"distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.357753 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.357729 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-model-cache\") pod \"distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.357814 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.357785 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-tmp-dir\") pod \"distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.357960 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.357823 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npn2l\" (UniqueName: \"kubernetes.io/projected/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-kube-api-access-npn2l\") pod \"distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.358030 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.357960 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-dshm\") pod \"distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.414697 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.414655 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:47.459286 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.459252 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-tls-certs\") pod \"distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.459507 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.459302 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-home\") pod \"distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.459507 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.459375 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-model-cache\") pod \"distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.459507 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.459418 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-tmp-dir\") pod \"distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.459507 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.459452 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npn2l\" (UniqueName: \"kubernetes.io/projected/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-kube-api-access-npn2l\") pod \"distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.459687 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.459512 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-dshm\") pod \"distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.459855 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.459830 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-home\") pod \"distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.459965 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.459868 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-tmp-dir\") pod \"distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.459965 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.459876 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-model-cache\") pod \"distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.461742 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.461713 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-dshm\") pod \"distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.461881 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.461862 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-tls-certs\") pod \"distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.519313 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.519231 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npn2l\" (UniqueName: \"kubernetes.io/projected/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-kube-api-access-npn2l\") pod \"distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.587747 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.587711 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:47.990697 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:47.990664 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f"] Nov 03 21:08:47.996302 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:08:47.996266 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfae009e3_4ccc_46ef_895c_5ecc6ea4e599.slice/crio-9b70724036f76e090217737727c2f616ad3e69ea754db6104c01b8999555da71 WatchSource:0}: Error finding container 9b70724036f76e090217737727c2f616ad3e69ea754db6104c01b8999555da71: Status 404 returned error can't find the container with id 9b70724036f76e090217737727c2f616ad3e69ea754db6104c01b8999555da71 Nov 03 21:08:48.289524 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.289434 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" event={"ID":"fae009e3-4ccc-46ef-895c-5ecc6ea4e599","Type":"ContainerStarted","Data":"9186ab3aaac1096ee0dc3e212754aaef3d9b2c38a0ccec92bd791d20a973697a"} Nov 03 21:08:48.289524 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.289477 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" event={"ID":"fae009e3-4ccc-46ef-895c-5ecc6ea4e599","Type":"ContainerStarted","Data":"9b70724036f76e090217737727c2f616ad3e69ea754db6104c01b8999555da71"} Nov 03 21:08:48.402496 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.402444 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" podStartSLOduration=1.402430429 podStartE2EDuration="1.402430429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:48.399955013 +0000 UTC m=+1277.560027285" watchObservedRunningTime="2026-11-03 21:08:48.402430429 +0000 UTC m=+1277.562502701" Nov 03 21:08:48.503197 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.503159 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr"] Nov 03 21:08:48.507255 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.507226 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.535158 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.535123 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw"] Nov 03 21:08:48.540752 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.540674 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.565807 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.565764 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr"] Nov 03 21:08:48.586386 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.586342 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.586557 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.586412 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.586557 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.586508 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.586645 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.586581 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.586645 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.586609 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.586727 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.586652 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.586787 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.586764 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brfh6\" (UniqueName: \"kubernetes.io/projected/5ece95b4-8e44-47ee-9719-e2066b5f81a1-kube-api-access-brfh6\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.586821 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.586804 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.586944 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.586893 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ncv\" (UniqueName: \"kubernetes.io/projected/c368337d-2d74-4904-a9c2-4199a2aac58f-kube-api-access-s4ncv\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.586986 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.586972 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.587024 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.587001 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.587081 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.587065 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.587116 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.587106 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.587206 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.587191 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.587244 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.587234 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.587277 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.587255 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.587324 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.587313 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c368337d-2d74-4904-a9c2-4199a2aac58f-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.587358 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.587332 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.587981 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.587968 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:48.613300 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.613262 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw"] Nov 03 21:08:48.688086 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688044 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.688086 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688084 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.688342 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688213 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.688342 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688276 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brfh6\" (UniqueName: \"kubernetes.io/projected/5ece95b4-8e44-47ee-9719-e2066b5f81a1-kube-api-access-brfh6\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.688342 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688307 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.688506 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688366 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ncv\" (UniqueName: \"kubernetes.io/projected/c368337d-2d74-4904-a9c2-4199a2aac58f-kube-api-access-s4ncv\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.688506 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688411 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.688506 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688438 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.688506 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688483 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.688506 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688493 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.688751 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688532 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.688751 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688612 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.688751 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688653 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.688751 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688677 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.688751 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688729 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c368337d-2d74-4904-a9c2-4199a2aac58f-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.688751 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688735 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.689085 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688758 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.689085 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688818 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.689085 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688821 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.689085 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688848 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.689085 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688944 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.689085 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.688979 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.689085 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.689044 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.689439 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.689141 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.689439 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.689251 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.689439 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.689321 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.689577 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.689491 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.689817 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.689790 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c368337d-2d74-4904-a9c2-4199a2aac58f-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.691243 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.691214 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.691338 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.691252 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.691640 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.691622 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.691677 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.691665 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.749958 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.749917 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.750131 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.750013 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.750195 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.750143 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ncv\" (UniqueName: \"kubernetes.io/projected/c368337d-2d74-4904-a9c2-4199a2aac58f-kube-api-access-s4ncv\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.750259 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.750242 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brfh6\" (UniqueName: \"kubernetes.io/projected/5ece95b4-8e44-47ee-9719-e2066b5f81a1-kube-api-access-brfh6\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.801040 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.800954 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822"] Nov 03 21:08:48.806306 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.806279 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.823199 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.823166 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:48.859977 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.859946 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-320f60-openshift-default-dockercfg-jct6g\"" Nov 03 21:08:48.861144 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.861114 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:48.863826 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.863800 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822"] Nov 03 21:08:48.892667 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.892578 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-envoy\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.892667 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.892638 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faeed472-64ed-40e2-aee1-0703462ccbe1-istiod-ca-cert\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.892864 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.892769 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-credential-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.892864 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.892804 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-token\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.893006 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.892991 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-workload-certs\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.893066 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.893021 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-data\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.893066 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.893047 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddbfn\" (UniqueName: \"kubernetes.io/projected/faeed472-64ed-40e2-aee1-0703462ccbe1-kube-api-access-ddbfn\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.893161 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.893135 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-workload-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.893214 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.893161 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-podinfo\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.993992 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.993958 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-data\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.993992 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.993995 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddbfn\" (UniqueName: \"kubernetes.io/projected/faeed472-64ed-40e2-aee1-0703462ccbe1-kube-api-access-ddbfn\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.994218 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.994090 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-workload-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.994218 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.994115 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-podinfo\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.994218 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.994198 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-envoy\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.994399 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.994232 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faeed472-64ed-40e2-aee1-0703462ccbe1-istiod-ca-cert\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.994399 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.994284 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-credential-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.994399 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.994312 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-token\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.994611 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.994411 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-data\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.994611 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.994453 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-workload-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.994611 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.994527 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-workload-certs\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.994761 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.994726 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-credential-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.994957 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.994897 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-workload-certs\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.995339 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.995316 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faeed472-64ed-40e2-aee1-0703462ccbe1-istiod-ca-cert\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.997127 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.997107 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-envoy\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:48.997456 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:48.997436 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-podinfo\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:49.055421 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:49.055339 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-token\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:49.055421 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:49.055382 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddbfn\" (UniqueName: \"kubernetes.io/projected/faeed472-64ed-40e2-aee1-0703462ccbe1-kube-api-access-ddbfn\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-s5822\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:49.121839 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:49.121804 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:49.266051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:49.266014 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr"] Nov 03 21:08:49.275030 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:49.269838 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw"] Nov 03 21:08:49.293531 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:49.293491 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:08:49.293670 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:49.293654 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:08:49.293874 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:49.293858 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:08:49.298614 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:49.297983 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:08:49.298614 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:49.298108 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:08:49.298614 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:49.298212 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:08:49.307711 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:49.307652 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" event={"ID":"c368337d-2d74-4904-a9c2-4199a2aac58f","Type":"ContainerStarted","Data":"75671f78972c5f5da25856669bfc6d6fe5ef5a470b994ec87725405360af9b36"} Nov 03 21:08:49.501844 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:49.501806 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822"] Nov 03 21:08:49.507150 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:08:49.507120 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaeed472_64ed_40e2_aee1_0703462ccbe1.slice/crio-c9d49b0efb494815a39d1c4623a707afdcfb78c7f507d907004985a04255b01d WatchSource:0}: Error finding container c9d49b0efb494815a39d1c4623a707afdcfb78c7f507d907004985a04255b01d: Status 404 returned error can't find the container with id c9d49b0efb494815a39d1c4623a707afdcfb78c7f507d907004985a04255b01d Nov 03 21:08:49.510095 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:49.510058 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:08:49.510218 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:49.510144 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:08:49.510218 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:49.510185 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:08:50.368205 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:50.368170 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" event={"ID":"5ece95b4-8e44-47ee-9719-e2066b5f81a1","Type":"ContainerStarted","Data":"490dee731b9930b5ecb4f5a600121efc94e62de4a3b848259cbbd4f243752ad0"} Nov 03 21:08:50.368388 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:50.368225 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" event={"ID":"5ece95b4-8e44-47ee-9719-e2066b5f81a1","Type":"ContainerStarted","Data":"971ce7749e57a9d25f28359827b25988a25a93990fca42b5c4ab61d8a25995d2"} Nov 03 21:08:50.370641 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:50.370606 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" event={"ID":"faeed472-64ed-40e2-aee1-0703462ccbe1","Type":"ContainerStarted","Data":"f44154bae9ca2ef65a260cb15365c81b744d8605a802c0ce6e42c133dc762d14"} Nov 03 21:08:50.370776 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:50.370658 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" event={"ID":"faeed472-64ed-40e2-aee1-0703462ccbe1","Type":"ContainerStarted","Data":"c9d49b0efb494815a39d1c4623a707afdcfb78c7f507d907004985a04255b01d"} Nov 03 21:08:50.372209 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:50.372181 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" event={"ID":"c368337d-2d74-4904-a9c2-4199a2aac58f","Type":"ContainerStarted","Data":"c78d2c1444429823597aa3b46dab57287b51a0b9de5fa4899930368e7cf0a427"} Nov 03 21:08:50.663728 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:50.663600 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" podStartSLOduration=2.663580756 podStartE2EDuration="2.663580756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:50.529052313 +0000 UTC m=+1279.689124586" watchObservedRunningTime="2026-11-03 21:08:50.663580756 +0000 UTC m=+1279.823653031" Nov 03 21:08:50.792393 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:50.790076 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" podStartSLOduration=2.790052712 podStartE2EDuration="2.790052712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:50.663305713 +0000 UTC m=+1279.823377986" watchObservedRunningTime="2026-11-03 21:08:50.790052712 +0000 UTC m=+1279.950124985" Nov 03 21:08:50.834655 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:50.834621 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:50.837251 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:50.837219 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 21:08:50.856639 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:50.856592 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" podUID="5ece95b4-8e44-47ee-9719-e2066b5f81a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 21:08:50.862954 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:50.862920 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:50.867248 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:50.867068 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 21:08:50.867248 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:50.867125 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" podUID="c368337d-2d74-4904-a9c2-4199a2aac58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 21:08:50.965951 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:50.960493 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:51.124287 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:51.124254 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:51.124626 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:51.124605 4963 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-s5822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 21:08:51.124780 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:51.124754 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" podUID="faeed472-64ed-40e2-aee1-0703462ccbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 21:08:51.838349 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:51.838307 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 21:08:51.838963 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:51.838928 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" podUID="5ece95b4-8e44-47ee-9719-e2066b5f81a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 21:08:51.862108 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:51.862071 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 21:08:51.862292 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:51.862132 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" podUID="c368337d-2d74-4904-a9c2-4199a2aac58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 21:08:52.144982 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:52.144300 4963 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-s5822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 21:08:52.144982 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:52.144357 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" podUID="faeed472-64ed-40e2-aee1-0703462ccbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 21:08:52.829930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:52.829341 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 21:08:52.829930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:52.829415 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" podUID="5ece95b4-8e44-47ee-9719-e2066b5f81a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 21:08:52.862080 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:52.862042 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 21:08:52.862490 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:52.862119 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" podUID="c368337d-2d74-4904-a9c2-4199a2aac58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 21:08:53.159994 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:53.151042 4963 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-s5822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 21:08:53.159994 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:53.151145 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" podUID="faeed472-64ed-40e2-aee1-0703462ccbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 21:08:53.826602 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:53.826565 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 21:08:53.826895 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:53.826865 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" podUID="5ece95b4-8e44-47ee-9719-e2066b5f81a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 21:08:53.862770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:53.862234 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 21:08:53.862770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:53.862294 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" podUID="c368337d-2d74-4904-a9c2-4199a2aac58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 21:08:54.125438 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:54.125328 4963 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-s5822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 21:08:54.125708 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:54.125672 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" podUID="faeed472-64ed-40e2-aee1-0703462ccbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 21:08:54.823930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:54.823529 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 21:08:54.823930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:54.823600 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" podUID="5ece95b4-8e44-47ee-9719-e2066b5f81a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 21:08:54.861991 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:54.861952 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 21:08:54.862168 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:54.862002 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" podUID="c368337d-2d74-4904-a9c2-4199a2aac58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 21:08:55.125497 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:55.125405 4963 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-s5822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 21:08:55.126124 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:55.126083 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" podUID="faeed472-64ed-40e2-aee1-0703462ccbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 21:08:55.824052 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:55.824009 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 21:08:55.824231 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:55.824079 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" podUID="5ece95b4-8e44-47ee-9719-e2066b5f81a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 21:08:55.862290 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:55.862249 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 21:08:55.862502 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:55.862316 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" podUID="c368337d-2d74-4904-a9c2-4199a2aac58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 21:08:55.884104 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:55.884063 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:56.122593 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:56.122492 4963 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-s5822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 21:08:56.122593 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:56.122564 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" podUID="faeed472-64ed-40e2-aee1-0703462ccbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 21:08:56.867353 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:56.867317 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:56.867881 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:56.867860 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:56.869244 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:56.869211 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:08:56.938444 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:56.938411 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:56.939277 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:56.939244 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:56.940672 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:56.940641 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:08:57.032550 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:57.032489 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" podStartSLOduration=9.032474417 podStartE2EDuration="9.032474417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:50.81562708 +0000 UTC m=+1279.975699368" watchObservedRunningTime="2026-11-03 21:08:57.032474417 +0000 UTC m=+1286.192546689" Nov 03 21:08:57.355272 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:57.355048 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:57.356629 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:57.356558 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:57.359788 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:57.359763 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:08:57.588322 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:57.588285 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:57.602721 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:57.602691 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:08:58.512989 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:08:58.512954 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:09:00.411232 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:00.411178 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:00.590246 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:00.590214 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:05.056541 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:05.056497 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:09.116066 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:09.116007 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" podUID="a3f5464f-5d94-499e-9fee-7553aa1e78a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:09.218064 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:09.218007 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" podUID="1edcbec2-0930-4b49-b14a-2d21568b5318" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:09.376206 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:09.376065 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" podUID="cf9455eb-457d-4fec-aca7-f55202209d2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 21:09:09.427065 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:09.427014 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" podUID="fa8baac2-125d-42f6-8449-5fd590bf4b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:09.502085 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:09.502024 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" podUID="fae009e3-4ccc-46ef-895c-5ecc6ea4e599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 21:09:09.654397 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:09.654273 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" podUID="b48999ea-6a3b-427c-a22b-621a5cc997db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:09.867939 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:09.866003 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" podUID="662bceeb-beae-4a4f-b08a-08706bed2d2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:09.971177 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:09.971050 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" podUID="4524d0a6-727b-47a9-b3b9-e83e5cf3a52d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:09.981280 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:09.981236 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" podUID="7bdfcd76-3dd3-46fd-a382-0e40e648aa17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:10.118070 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:10.118015 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" podUID="a3f5464f-5d94-499e-9fee-7553aa1e78a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:10.120199 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:10.119155 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:09:10.220571 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:10.218417 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" podUID="1edcbec2-0930-4b49-b14a-2d21568b5318" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:10.220571 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:10.219127 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:09:10.387004 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:10.385103 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" podUID="cf9455eb-457d-4fec-aca7-f55202209d2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:10.387004 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:10.385803 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:09:10.434181 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:10.434131 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:10.511814 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:10.511772 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" podUID="fae009e3-4ccc-46ef-895c-5ecc6ea4e599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 21:09:10.512803 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:10.512777 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:09:10.835589 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:10.835559 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 21:09:11.222069 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:11.222008 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" podUID="1edcbec2-0930-4b49-b14a-2d21568b5318" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:11.429928 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:11.426125 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" podUID="cf9455eb-457d-4fec-aca7-f55202209d2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:11.462007 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:11.461976 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 21:09:11.464061 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:11.463717 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 21:09:11.464897 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:11.464764 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 21:09:12.528421 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:09:12.528385 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:09:13.837862 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:13.837827 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:14.502004 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:14.501943 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:14.755726 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:14.755640 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:15.413155 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:15.413105 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:18.869144 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:18.869098 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:19.051540 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:19.051504 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:20.100887 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:20.100854 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:20.881318 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:20.881122 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:22.264166 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:22.264134 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:22.448318 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:22.448255 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:24.852567 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:24.852539 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:28.375023 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:28.374986 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:28.657662 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:28.657571 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:28.853599 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:28.853559 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:29.428595 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:29.428560 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:30.715930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:30.715127 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" podUID="b48999ea-6a3b-427c-a22b-621a5cc997db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:30.958245 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:30.958194 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" podUID="4524d0a6-727b-47a9-b3b9-e83e5cf3a52d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:34.204776 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:34.204734 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:38.719725 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:38.719677 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:39.585735 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:39.585694 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:39.895215 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:39.895136 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:42.536939 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:09:42.536883 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:09:43.368015 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:43.367968 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:45.955298 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:45.955258 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:46.629341 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:46.629223 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:46.915859 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:46.915740 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:48.044199 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:48.044158 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:48.226359 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:48.226320 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:48.411617 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:48.411522 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:51.341653 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:51.341621 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:51.542313 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:51.542280 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:53.044794 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:53.044759 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:53.769730 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:53.769701 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:55.411674 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:55.411613 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:55.599732 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:55.599673 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:55.786978 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:55.786945 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:57.918483 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:57.918446 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:58.688632 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:58.688593 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:58.925465 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:58.925428 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:59.123096 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:59.123059 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:59.923930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:09:59.923860 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:00.703013 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:00.702974 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:01.170853 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:01.170760 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:03.196733 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:03.196699 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:05.114533 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:05.114499 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:05.320547 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:05.320504 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:05.509617 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:05.509581 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:07.084529 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:07.084501 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:07.299302 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:07.299269 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:08.221235 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:08.221196 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:10.693931 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:10.693767 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:10.884516 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:10.884485 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:12.529064 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:10:12.529021 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:10:12.651587 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:12.651553 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:12.835837 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:12.835743 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:14.188765 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:14.188728 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:17.415715 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:17.415681 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:18.656104 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:18.656072 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:18.839793 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:18.839760 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:21.931408 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:21.931351 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:26.235192 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.235153 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4"] Nov 03 21:10:26.239553 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.239532 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:10:26.291945 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.291892 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84819654-ad13-4357-be29-5f8f17e05d1e-metrics-certs\") pod \"maas-api-e2e-api-a-145cc5-76df675754-gq4v4\" (UID: \"84819654-ad13-4357-be29-5f8f17e05d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:10:26.292147 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.291963 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84819654-ad13-4357-be29-5f8f17e05d1e-maas-api-tls\") pod \"maas-api-e2e-api-a-145cc5-76df675754-gq4v4\" (UID: \"84819654-ad13-4357-be29-5f8f17e05d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:10:26.292147 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.292128 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzqqh\" (UniqueName: \"kubernetes.io/projected/84819654-ad13-4357-be29-5f8f17e05d1e-kube-api-access-zzqqh\") pod \"maas-api-e2e-api-a-145cc5-76df675754-gq4v4\" (UID: \"84819654-ad13-4357-be29-5f8f17e05d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:10:26.292261 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.292224 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84819654-ad13-4357-be29-5f8f17e05d1e-openshift-service-ca\") pod \"maas-api-e2e-api-a-145cc5-76df675754-gq4v4\" (UID: \"84819654-ad13-4357-be29-5f8f17e05d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:10:26.336516 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.336480 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-145cc5-serving-cert\"" Nov 03 21:10:26.339943 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.339914 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4"] Nov 03 21:10:26.393202 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.393166 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84819654-ad13-4357-be29-5f8f17e05d1e-maas-api-tls\") pod \"maas-api-e2e-api-a-145cc5-76df675754-gq4v4\" (UID: \"84819654-ad13-4357-be29-5f8f17e05d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:10:26.393412 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.393263 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzqqh\" (UniqueName: \"kubernetes.io/projected/84819654-ad13-4357-be29-5f8f17e05d1e-kube-api-access-zzqqh\") pod \"maas-api-e2e-api-a-145cc5-76df675754-gq4v4\" (UID: \"84819654-ad13-4357-be29-5f8f17e05d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:10:26.393412 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.393313 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84819654-ad13-4357-be29-5f8f17e05d1e-openshift-service-ca\") pod \"maas-api-e2e-api-a-145cc5-76df675754-gq4v4\" (UID: \"84819654-ad13-4357-be29-5f8f17e05d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:10:26.393412 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:10:26.393336 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-serving-cert: secret "maas-api-e2e-api-a-145cc5-serving-cert" not found Nov 03 21:10:26.393585 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:10:26.393436 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84819654-ad13-4357-be29-5f8f17e05d1e-maas-api-tls podName:84819654-ad13-4357-be29-5f8f17e05d1e nodeName:}" failed. No retries permitted until 2026-11-03 21:10:26.893413617 +0000 UTC m=+1376.053485882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84819654-ad13-4357-be29-5f8f17e05d1e-maas-api-tls") pod "maas-api-e2e-api-a-145cc5-76df675754-gq4v4" (UID: "84819654-ad13-4357-be29-5f8f17e05d1e") : secret "maas-api-e2e-api-a-145cc5-serving-cert" not found Nov 03 21:10:26.393585 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.393568 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84819654-ad13-4357-be29-5f8f17e05d1e-metrics-certs\") pod \"maas-api-e2e-api-a-145cc5-76df675754-gq4v4\" (UID: \"84819654-ad13-4357-be29-5f8f17e05d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:10:26.394132 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.394109 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84819654-ad13-4357-be29-5f8f17e05d1e-openshift-service-ca\") pod \"maas-api-e2e-api-a-145cc5-76df675754-gq4v4\" (UID: \"84819654-ad13-4357-be29-5f8f17e05d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:10:26.396140 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.396121 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84819654-ad13-4357-be29-5f8f17e05d1e-metrics-certs\") pod \"maas-api-e2e-api-a-145cc5-76df675754-gq4v4\" (UID: \"84819654-ad13-4357-be29-5f8f17e05d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:10:26.452441 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.452411 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzqqh\" (UniqueName: \"kubernetes.io/projected/84819654-ad13-4357-be29-5f8f17e05d1e-kube-api-access-zzqqh\") pod \"maas-api-e2e-api-a-145cc5-76df675754-gq4v4\" (UID: \"84819654-ad13-4357-be29-5f8f17e05d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:10:26.465833 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.465801 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:26.655211 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.655116 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:26.841855 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.841820 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:26.899290 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.899255 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84819654-ad13-4357-be29-5f8f17e05d1e-maas-api-tls\") pod \"maas-api-e2e-api-a-145cc5-76df675754-gq4v4\" (UID: \"84819654-ad13-4357-be29-5f8f17e05d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:10:26.901786 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:26.901764 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84819654-ad13-4357-be29-5f8f17e05d1e-maas-api-tls\") pod \"maas-api-e2e-api-a-145cc5-76df675754-gq4v4\" (UID: \"84819654-ad13-4357-be29-5f8f17e05d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:10:27.153372 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:27.153336 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:10:27.533558 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:27.533521 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4"] Nov 03 21:10:27.538788 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:10:27.538761 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84819654_ad13_4357_be29_5f8f17e05d1e.slice/crio-07139ae99bdb7711471d04784bef86df660b7bbe56fb294d05249df42c5108af WatchSource:0}: Error finding container 07139ae99bdb7711471d04784bef86df660b7bbe56fb294d05249df42c5108af: Status 404 returned error can't find the container with id 07139ae99bdb7711471d04784bef86df660b7bbe56fb294d05249df42c5108af Nov 03 21:10:27.540237 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:27.540219 4963 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:10:28.428712 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:28.428672 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" event={"ID":"84819654-ad13-4357-be29-5f8f17e05d1e","Type":"ContainerStarted","Data":"07139ae99bdb7711471d04784bef86df660b7bbe56fb294d05249df42c5108af"} Nov 03 21:10:28.704128 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:28.704089 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:29.437617 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:29.437580 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" event={"ID":"84819654-ad13-4357-be29-5f8f17e05d1e","Type":"ContainerStarted","Data":"7e6f10cecf94bf8343656205ba7c2f961950c54a36b56cc7acb26a769eaeb63c"} Nov 03 21:10:29.437800 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:29.437715 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:10:29.550715 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:29.550661 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" podStartSLOduration=2.645602251 podStartE2EDuration="3.55064918s" totalImagesPullingTime="905.046929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:29.550313649 +0000 UTC m=+1378.710385921" watchObservedRunningTime="2026-11-03 21:10:29.55064918 +0000 UTC m=+1378.710721451" Nov 03 21:10:31.411232 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:31.411193 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:32.544332 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.544295 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs"] Nov 03 21:10:32.548573 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.548557 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.603184 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.603150 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs"] Nov 03 21:10:32.646763 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.646724 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e84357\"/\"distinct-w4-e84357-kserve-self-signed-certs\"" Nov 03 21:10:32.670490 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.670457 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:32.671079 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.671058 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-dshm\") pod \"distinct-w4-e84357-kserve-58bd574449-8g8gs\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.671130 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.671094 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d38fac8-2315-4693-8dae-4e449356ee4c-tls-certs\") pod \"distinct-w4-e84357-kserve-58bd574449-8g8gs\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.671180 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.671168 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-model-cache\") pod \"distinct-w4-e84357-kserve-58bd574449-8g8gs\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.671258 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.671237 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-home\") pod \"distinct-w4-e84357-kserve-58bd574449-8g8gs\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.671348 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.671333 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt8hk\" (UniqueName: \"kubernetes.io/projected/7d38fac8-2315-4693-8dae-4e449356ee4c-kube-api-access-vt8hk\") pod \"distinct-w4-e84357-kserve-58bd574449-8g8gs\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.671429 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.671418 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-tmp-dir\") pod \"distinct-w4-e84357-kserve-58bd574449-8g8gs\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.772871 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.772835 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt8hk\" (UniqueName: \"kubernetes.io/projected/7d38fac8-2315-4693-8dae-4e449356ee4c-kube-api-access-vt8hk\") pod \"distinct-w4-e84357-kserve-58bd574449-8g8gs\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.773100 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.772949 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-tmp-dir\") pod \"distinct-w4-e84357-kserve-58bd574449-8g8gs\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.773100 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.773068 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-dshm\") pod \"distinct-w4-e84357-kserve-58bd574449-8g8gs\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.773100 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.773094 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d38fac8-2315-4693-8dae-4e449356ee4c-tls-certs\") pod \"distinct-w4-e84357-kserve-58bd574449-8g8gs\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.773276 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.773134 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-model-cache\") pod \"distinct-w4-e84357-kserve-58bd574449-8g8gs\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.773276 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.773169 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-home\") pod \"distinct-w4-e84357-kserve-58bd574449-8g8gs\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.773389 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.773351 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-tmp-dir\") pod \"distinct-w4-e84357-kserve-58bd574449-8g8gs\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.773955 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.773882 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-model-cache\") pod \"distinct-w4-e84357-kserve-58bd574449-8g8gs\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.774151 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.773950 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-home\") pod \"distinct-w4-e84357-kserve-58bd574449-8g8gs\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.776931 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.776886 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-dshm\") pod \"distinct-w4-e84357-kserve-58bd574449-8g8gs\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.795953 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.795871 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d38fac8-2315-4693-8dae-4e449356ee4c-tls-certs\") pod \"distinct-w4-e84357-kserve-58bd574449-8g8gs\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.831456 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.831415 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt8hk\" (UniqueName: \"kubernetes.io/projected/7d38fac8-2315-4693-8dae-4e449356ee4c-kube-api-access-vt8hk\") pod \"distinct-w4-e84357-kserve-58bd574449-8g8gs\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:32.861359 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:32.861325 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:33.289464 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:33.289427 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs"] Nov 03 21:10:33.294469 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:10:33.294440 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d38fac8_2315_4693_8dae_4e449356ee4c.slice/crio-50a0eab55d034dae100e133df62790f4e4a80b80b9b534f01185f75a39ec5406 WatchSource:0}: Error finding container 50a0eab55d034dae100e133df62790f4e4a80b80b9b534f01185f75a39ec5406: Status 404 returned error can't find the container with id 50a0eab55d034dae100e133df62790f4e4a80b80b9b534f01185f75a39ec5406 Nov 03 21:10:33.471589 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:33.471545 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" event={"ID":"7d38fac8-2315-4693-8dae-4e449356ee4c","Type":"ContainerStarted","Data":"c358e786cb0030bc7963220a79ec8f9e09f159d5bf56c24ffc7be69b5de5024d"} Nov 03 21:10:33.471768 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:33.471614 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" event={"ID":"7d38fac8-2315-4693-8dae-4e449356ee4c","Type":"ContainerStarted","Data":"50a0eab55d034dae100e133df62790f4e4a80b80b9b534f01185f75a39ec5406"} Nov 03 21:10:33.585715 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:33.585597 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" podStartSLOduration=1.585580866 podStartE2EDuration="1.585580866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:33.582847881 +0000 UTC m=+1382.742920153" watchObservedRunningTime="2026-11-03 21:10:33.585580866 +0000 UTC m=+1382.745653137" Nov 03 21:10:33.861709 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:33.861628 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:35.450008 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:35.449977 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:10:35.917336 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:35.917223 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:40.636226 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:40.636089 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:42.528790 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:10:42.528745 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:10:42.862032 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:42.861931 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:42.874494 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:42.874467 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:43.542848 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:43.542818 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 21:10:47.191211 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:47.191175 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:47.374743 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:47.374712 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:49.809738 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:49.809703 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:54.406731 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:54.406693 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:58.887406 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:10:58.887364 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:03.417538 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:03.417495 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:08.014434 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:08.014391 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:12.529247 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:11:12.529209 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:11:12.546449 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:12.546421 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:17.316108 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:17.316062 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:18.887389 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:18.887353 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:19.848229 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:19.848192 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:20.076820 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.076787 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:20.298185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.298149 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx"] Nov 03 21:11:20.302559 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.302536 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.400136 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.400105 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-32f6e43c\"/\"kube-root-ca.crt\"" Nov 03 21:11:20.400136 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.400114 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-32f6e43c\"/\"openshift-service-ca.crt\"" Nov 03 21:11:20.400401 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.400199 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-32f6e43c\"/\"default-dockercfg-ndgm2\"" Nov 03 21:11:20.400401 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.400334 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-32f6e43c\"/\"e2e-lifecycle-model-32f6e43c-kserve-self-signed-certs\"" Nov 03 21:11:20.403417 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.403391 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx"] Nov 03 21:11:20.405982 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.405959 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-model-cache\") pod \"e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.406106 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.406032 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq76r\" (UniqueName: \"kubernetes.io/projected/f315250f-0dc8-4b98-bb9a-9d998e62ac01-kube-api-access-lq76r\") pod \"e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.406156 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.406110 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-tmp-dir\") pod \"e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.406206 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.406157 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f315250f-0dc8-4b98-bb9a-9d998e62ac01-tls-certs\") pod \"e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.406318 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.406306 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-dshm\") pod \"e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.406356 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.406325 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-home\") pod \"e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.440499 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.440467 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:20.507289 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.507247 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-dshm\") pod \"e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.507289 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.507288 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-home\") pod \"e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.507568 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.507338 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-model-cache\") pod \"e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.507568 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.507360 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq76r\" (UniqueName: \"kubernetes.io/projected/f315250f-0dc8-4b98-bb9a-9d998e62ac01-kube-api-access-lq76r\") pod \"e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.507568 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.507392 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-tmp-dir\") pod \"e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.507784 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.507746 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f315250f-0dc8-4b98-bb9a-9d998e62ac01-tls-certs\") pod \"e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.507869 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.507821 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-model-cache\") pod \"e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.507869 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.507851 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-tmp-dir\") pod \"e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.507997 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.507898 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-home\") pod \"e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.510234 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.510208 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-dshm\") pod \"e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.511076 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.511052 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f315250f-0dc8-4b98-bb9a-9d998e62ac01-tls-certs\") pod \"e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.567416 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.567331 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq76r\" (UniqueName: \"kubernetes.io/projected/f315250f-0dc8-4b98-bb9a-9d998e62ac01-kube-api-access-lq76r\") pod \"e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:20.615934 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:20.615860 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:21.010758 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:21.010676 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx"] Nov 03 21:11:21.017005 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:11:21.016970 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf315250f_0dc8_4b98_bb9a_9d998e62ac01.slice/crio-0025c8a6f3fff9b71c2f62afa3e364775d5f3c3bc04e7f3a998741a9dc714010 WatchSource:0}: Error finding container 0025c8a6f3fff9b71c2f62afa3e364775d5f3c3bc04e7f3a998741a9dc714010: Status 404 returned error can't find the container with id 0025c8a6f3fff9b71c2f62afa3e364775d5f3c3bc04e7f3a998741a9dc714010 Nov 03 21:11:21.824849 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:21.824812 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" event={"ID":"f315250f-0dc8-4b98-bb9a-9d998e62ac01","Type":"ContainerStarted","Data":"882fbddf3e2e325f6bc9a26268f0999404e8a02f73a5b77259e6ef0bff10c539"} Nov 03 21:11:21.824849 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:21.824855 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" event={"ID":"f315250f-0dc8-4b98-bb9a-9d998e62ac01","Type":"ContainerStarted","Data":"0025c8a6f3fff9b71c2f62afa3e364775d5f3c3bc04e7f3a998741a9dc714010"} Nov 03 21:11:21.952349 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:21.952291 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" podStartSLOduration=1.952275234 podStartE2EDuration="1.952275234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:21.941514874 +0000 UTC m=+1431.101587143" watchObservedRunningTime="2026-11-03 21:11:21.952275234 +0000 UTC m=+1431.112347507" Nov 03 21:11:22.010637 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:22.010598 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:22.616724 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:22.616681 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:23.678192 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:23.678159 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:23.861157 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:23.861126 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:25.693513 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:25.693474 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:26.840736 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:26.840702 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:27.032654 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:27.032572 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:27.215054 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:27.215017 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:29.077022 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:29.076988 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:30.616743 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:30.616706 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:30.629864 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:30.629834 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:30.910196 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:30.910120 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:11:31.617063 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:31.617026 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:32.954797 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:32.954759 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:33.444176 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:33.444137 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:33.631821 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:33.631775 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:36.316527 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:36.316489 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:37.853404 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:37.853351 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:37.946706 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:37.946671 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5"] Nov 03 21:11:37.952641 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:37.952616 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.041108 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.041061 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-home\") pod \"distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.041336 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.041309 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-dshm\") pod \"distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.041509 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.041483 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce93966-0d0b-4b1d-aa96-16315a7467b5-tls-certs\") pod \"distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.041760 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.041720 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-model-cache\") pod \"distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.041899 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.041770 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-tmp-dir\") pod \"distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.041899 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.041796 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr24z\" (UniqueName: \"kubernetes.io/projected/8ce93966-0d0b-4b1d-aa96-16315a7467b5-kube-api-access-tr24z\") pod \"distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.052005 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.051978 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e84357\"/\"distinct-2-w4-e84357-kserve-self-signed-certs\"" Nov 03 21:11:38.055434 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.055405 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:38.055568 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.055549 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5"] Nov 03 21:11:38.150551 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.150457 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-home\") pod \"distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.150890 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.150867 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-dshm\") pod \"distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.151074 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.151059 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce93966-0d0b-4b1d-aa96-16315a7467b5-tls-certs\") pod \"distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.151279 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.151264 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-model-cache\") pod \"distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.151376 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.151364 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-tmp-dir\") pod \"distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.151463 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.151451 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr24z\" (UniqueName: \"kubernetes.io/projected/8ce93966-0d0b-4b1d-aa96-16315a7467b5-kube-api-access-tr24z\") pod \"distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.152286 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.152258 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-home\") pod \"distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.154167 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.152952 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-model-cache\") pod \"distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.175497 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.175462 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-tmp-dir\") pod \"distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.177096 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.177069 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-dshm\") pod \"distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.182555 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.182527 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce93966-0d0b-4b1d-aa96-16315a7467b5-tls-certs\") pod \"distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.211816 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.211646 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr24z\" (UniqueName: \"kubernetes.io/projected/8ce93966-0d0b-4b1d-aa96-16315a7467b5-kube-api-access-tr24z\") pod \"distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.254794 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.254757 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:38.266741 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.266694 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:38.720582 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.720265 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:38.747387 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.747330 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5"] Nov 03 21:11:38.955540 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.955495 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" event={"ID":"8ce93966-0d0b-4b1d-aa96-16315a7467b5","Type":"ContainerStarted","Data":"297be8dc97ece4b5b7f19d5ccea8ac0842f5531b629e0e338fdf4c03688d8fa6"} Nov 03 21:11:38.956066 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:38.955555 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" event={"ID":"8ce93966-0d0b-4b1d-aa96-16315a7467b5","Type":"ContainerStarted","Data":"cfd0ecb19712e6dfdddf208a1cf948315c3c09a176be5721d7189ac4b7f146ce"} Nov 03 21:11:39.120669 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:39.120540 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" podStartSLOduration=2.120518412 podStartE2EDuration="2.120518412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:39.116979208 +0000 UTC m=+1448.277051482" watchObservedRunningTime="2026-11-03 21:11:39.120518412 +0000 UTC m=+1448.280590685" Nov 03 21:11:39.267333 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:39.267298 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:39.271225 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:39.271193 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:39.946307 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:39.946267 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:41.319412 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:41.319381 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:42.528691 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:11:42.528655 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:11:43.130453 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:43.130421 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:43.390412 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:43.390330 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:43.578766 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:43.578734 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:45.712054 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:45.712019 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:45.903874 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:45.903838 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:46.337198 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:46.337163 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:46.614805 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:46.614725 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:48.268941 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:48.267435 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:48.287280 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:48.287245 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:49.030867 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:49.030834 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 21:11:50.784810 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:50.784777 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:50.988212 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:50.988179 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:55.309413 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:55.309371 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:59.914697 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:11:59.914660 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:04.141918 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.141865 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg"] Nov 03 21:12:04.146521 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.146491 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:12:04.197395 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.197353 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6ad223-serving-cert\"" Nov 03 21:12:04.250245 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.250198 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg"] Nov 03 21:12:04.276192 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.276160 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab7cfd1-1ae1-4858-8b51-ed89214a47de-openshift-service-ca\") pod \"maas-api-e2e-api-b-6ad223-569f4d679b-wbspg\" (UID: \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:12:04.276402 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.276210 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab7cfd1-1ae1-4858-8b51-ed89214a47de-maas-api-tls\") pod \"maas-api-e2e-api-b-6ad223-569f4d679b-wbspg\" (UID: \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:12:04.276526 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.276501 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab7cfd1-1ae1-4858-8b51-ed89214a47de-metrics-certs\") pod \"maas-api-e2e-api-b-6ad223-569f4d679b-wbspg\" (UID: \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:12:04.276622 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.276606 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm6r9\" (UniqueName: \"kubernetes.io/projected/bab7cfd1-1ae1-4858-8b51-ed89214a47de-kube-api-access-lm6r9\") pod \"maas-api-e2e-api-b-6ad223-569f4d679b-wbspg\" (UID: \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:12:04.360619 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.360586 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:04.378858 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.378813 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab7cfd1-1ae1-4858-8b51-ed89214a47de-metrics-certs\") pod \"maas-api-e2e-api-b-6ad223-569f4d679b-wbspg\" (UID: \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:12:04.379874 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.379819 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm6r9\" (UniqueName: \"kubernetes.io/projected/bab7cfd1-1ae1-4858-8b51-ed89214a47de-kube-api-access-lm6r9\") pod \"maas-api-e2e-api-b-6ad223-569f4d679b-wbspg\" (UID: \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:12:04.380140 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.379968 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab7cfd1-1ae1-4858-8b51-ed89214a47de-openshift-service-ca\") pod \"maas-api-e2e-api-b-6ad223-569f4d679b-wbspg\" (UID: \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:12:04.380140 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.380024 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab7cfd1-1ae1-4858-8b51-ed89214a47de-maas-api-tls\") pod \"maas-api-e2e-api-b-6ad223-569f4d679b-wbspg\" (UID: \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:12:04.380351 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:12:04.380309 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-serving-cert: secret "maas-api-e2e-api-b-6ad223-serving-cert" not found Nov 03 21:12:04.380436 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:12:04.380383 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bab7cfd1-1ae1-4858-8b51-ed89214a47de-maas-api-tls podName:bab7cfd1-1ae1-4858-8b51-ed89214a47de nodeName:}" failed. No retries permitted until 2026-11-03 21:12:04.880366724 +0000 UTC m=+1474.040438988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bab7cfd1-1ae1-4858-8b51-ed89214a47de-maas-api-tls") pod "maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" (UID: "bab7cfd1-1ae1-4858-8b51-ed89214a47de") : secret "maas-api-e2e-api-b-6ad223-serving-cert" not found Nov 03 21:12:04.381729 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.381697 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab7cfd1-1ae1-4858-8b51-ed89214a47de-openshift-service-ca\") pod \"maas-api-e2e-api-b-6ad223-569f4d679b-wbspg\" (UID: \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:12:04.387750 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.387708 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab7cfd1-1ae1-4858-8b51-ed89214a47de-metrics-certs\") pod \"maas-api-e2e-api-b-6ad223-569f4d679b-wbspg\" (UID: \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:12:04.445684 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.445642 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm6r9\" (UniqueName: \"kubernetes.io/projected/bab7cfd1-1ae1-4858-8b51-ed89214a47de-kube-api-access-lm6r9\") pod \"maas-api-e2e-api-b-6ad223-569f4d679b-wbspg\" (UID: \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:12:04.544504 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.544471 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:04.726655 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.726569 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:04.888514 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.888479 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab7cfd1-1ae1-4858-8b51-ed89214a47de-maas-api-tls\") pod \"maas-api-e2e-api-b-6ad223-569f4d679b-wbspg\" (UID: \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:12:04.891168 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:04.891142 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab7cfd1-1ae1-4858-8b51-ed89214a47de-maas-api-tls\") pod \"maas-api-e2e-api-b-6ad223-569f4d679b-wbspg\" (UID: \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:12:05.061744 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:05.061652 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:12:05.439041 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:05.438949 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg"] Nov 03 21:12:05.444805 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:12:05.444777 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab7cfd1_1ae1_4858_8b51_ed89214a47de.slice/crio-30d04f54bdc530d89a68dae64e28d320dc8c5bed3334ed27d7f8542f0f780015 WatchSource:0}: Error finding container 30d04f54bdc530d89a68dae64e28d320dc8c5bed3334ed27d7f8542f0f780015: Status 404 returned error can't find the container with id 30d04f54bdc530d89a68dae64e28d320dc8c5bed3334ed27d7f8542f0f780015 Nov 03 21:12:06.154922 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:06.154870 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" event={"ID":"bab7cfd1-1ae1-4858-8b51-ed89214a47de","Type":"ContainerStarted","Data":"30d04f54bdc530d89a68dae64e28d320dc8c5bed3334ed27d7f8542f0f780015"} Nov 03 21:12:07.169244 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:07.169204 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" event={"ID":"bab7cfd1-1ae1-4858-8b51-ed89214a47de","Type":"ContainerStarted","Data":"0cc0c4b606292ba91263b54ce67f9a104bb10b042e6c9d6ea0bc79a110c0d18c"} Nov 03 21:12:07.171139 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:07.171114 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:12:07.283936 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:07.283843 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" podStartSLOduration=2.229686276 podStartE2EDuration="3.283828812s" totalImagesPullingTime="1.054142536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:07.283296577 +0000 UTC m=+1476.443368852" watchObservedRunningTime="2026-11-03 21:12:07.283828812 +0000 UTC m=+1476.443901084" Nov 03 21:12:08.984677 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:08.984639 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:12.528875 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:12:12.528790 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:12:13.467474 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:13.467436 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:14.188791 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:14.188759 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:12:16.231177 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.231140 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9"] Nov 03 21:12:16.235716 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.235684 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.244883 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.244861 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.245000 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.244894 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.245039 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.245026 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.245122 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.245104 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.245264 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.245241 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.245299 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.245285 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.245335 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.245320 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.245383 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.245369 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.245429 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.245416 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68n6k\" (UniqueName: \"kubernetes.io/projected/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-kube-api-access-68n6k\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.290700 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.290667 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e84357-openshift-default-dockercfg-t4w57\"" Nov 03 21:12:16.346042 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.345981 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9"] Nov 03 21:12:16.350743 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.350704 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.350955 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.350765 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.350955 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.350797 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.350955 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.350833 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.350955 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.350883 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68n6k\" (UniqueName: \"kubernetes.io/projected/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-kube-api-access-68n6k\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.351212 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.351029 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.351212 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.351069 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.351212 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.351143 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.351368 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.351213 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.353426 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.352641 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.353426 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.352976 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.354430 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.354403 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.355154 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.355131 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.355599 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.355577 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.361966 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.361941 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.365981 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.365951 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.410467 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.410426 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.412512 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.412482 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68n6k\" (UniqueName: \"kubernetes.io/projected/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-kube-api-access-68n6k\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.549977 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.549855 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:16.980268 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.980234 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9"] Nov 03 21:12:16.985243 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:12:16.985216 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe1449b_31f2_45a8_b054_d4dc8ec7e72d.slice/crio-48f1f5f08511a422c408f3219d0505efec58defdb490f71d082a0517f579519f WatchSource:0}: Error finding container 48f1f5f08511a422c408f3219d0505efec58defdb490f71d082a0517f579519f: Status 404 returned error can't find the container with id 48f1f5f08511a422c408f3219d0505efec58defdb490f71d082a0517f579519f Nov 03 21:12:16.988292 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.988252 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:12:16.988367 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.988344 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:12:16.988412 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:16.988387 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:12:17.241333 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:17.241246 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" event={"ID":"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d","Type":"ContainerStarted","Data":"ef1ec71e298fee1c3a71e831a4c80c2498cc1a16d0db94f20615b0e08874a9b5"} Nov 03 21:12:17.241333 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:17.241304 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" event={"ID":"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d","Type":"ContainerStarted","Data":"48f1f5f08511a422c408f3219d0505efec58defdb490f71d082a0517f579519f"} Nov 03 21:12:17.357621 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:17.357566 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" podStartSLOduration=1.357551957 podStartE2EDuration="1.357551957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:17.356661622 +0000 UTC m=+1486.516733894" watchObservedRunningTime="2026-11-03 21:12:17.357551957 +0000 UTC m=+1486.517624306" Nov 03 21:12:17.550419 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:17.550339 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:18.007699 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:18.007646 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:18.553213 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:18.552808 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:12:18.553213 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:18.552856 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" podUID="cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 21:12:19.551132 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:19.551094 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:12:19.551317 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:19.551147 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" podUID="cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 21:12:20.550591 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:20.550556 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:12:20.551010 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:20.550611 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" podUID="cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 21:12:21.550293 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:21.550259 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:12:21.550497 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:21.550314 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" podUID="cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 21:12:22.285926 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.281891 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv"] Nov 03 21:12:22.318383 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.318352 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:12:22.362340 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.360430 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv"] Nov 03 21:12:22.374282 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.373932 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3e92e3-serving-cert\"" Nov 03 21:12:22.407834 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.403844 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-metrics-certs\") pod \"maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv\" (UID: \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:12:22.407834 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.404036 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-maas-api-tls\") pod \"maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv\" (UID: \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:12:22.407834 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.404104 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-openshift-service-ca\") pod \"maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv\" (UID: \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:12:22.407834 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.404198 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm2bp\" (UniqueName: \"kubernetes.io/projected/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-kube-api-access-bm2bp\") pod \"maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv\" (UID: \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:12:22.506956 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.506853 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:22.507598 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.507474 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-openshift-service-ca\") pod \"maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv\" (UID: \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:12:22.507598 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.507584 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2bp\" (UniqueName: \"kubernetes.io/projected/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-kube-api-access-bm2bp\") pod \"maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv\" (UID: \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:12:22.507781 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.507649 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-metrics-certs\") pod \"maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv\" (UID: \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:12:22.507837 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.507821 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-maas-api-tls\") pod \"maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv\" (UID: \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:12:22.508556 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:12:22.508531 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-serving-cert: secret "maas-api-e2e-api-a-3e92e3-serving-cert" not found Nov 03 21:12:22.508673 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:12:22.508661 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-maas-api-tls podName:107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e nodeName:}" failed. No retries permitted until 2026-11-03 21:12:23.008623508 +0000 UTC m=+1492.168695775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-maas-api-tls") pod "maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" (UID: "107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e") : secret "maas-api-e2e-api-a-3e92e3-serving-cert" not found Nov 03 21:12:22.509052 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.509027 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-openshift-service-ca\") pod \"maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv\" (UID: \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:12:22.511591 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.511549 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-metrics-certs\") pod \"maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv\" (UID: \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:12:22.551028 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.550888 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:12:22.551028 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.551005 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" podUID="cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 21:12:22.575051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.575014 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm2bp\" (UniqueName: \"kubernetes.io/projected/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-kube-api-access-bm2bp\") pod \"maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv\" (UID: \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:12:22.704523 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.704408 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:22.899412 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:22.899326 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:23.017450 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:23.017412 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-maas-api-tls\") pod \"maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv\" (UID: \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:12:23.020629 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:23.020605 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-maas-api-tls\") pod \"maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv\" (UID: \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:12:23.029190 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:23.029159 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:12:23.434936 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:23.434887 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv"] Nov 03 21:12:23.439724 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:12:23.439693 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107dcc3b_5c84_4d4c_a5a4_697fe8b4e07e.slice/crio-3575eaf0fe4b0b39d7105a2d854ea99ea470a879d656ef65c854413edb33d1b9 WatchSource:0}: Error finding container 3575eaf0fe4b0b39d7105a2d854ea99ea470a879d656ef65c854413edb33d1b9: Status 404 returned error can't find the container with id 3575eaf0fe4b0b39d7105a2d854ea99ea470a879d656ef65c854413edb33d1b9 Nov 03 21:12:23.550471 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:23.550435 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 21:12:23.550643 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:23.550503 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" podUID="cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 21:12:24.341530 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:24.336712 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" event={"ID":"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e","Type":"ContainerStarted","Data":"3575eaf0fe4b0b39d7105a2d854ea99ea470a879d656ef65c854413edb33d1b9"} Nov 03 21:12:24.805484 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:24.805446 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:24.806094 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:24.806065 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:24.807077 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:24.807052 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 21:12:25.347013 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:25.346968 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" event={"ID":"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e","Type":"ContainerStarted","Data":"4e8890db5848c1bf81557302004c6b03c05b605b66905851457022895fbf1e85"} Nov 03 21:12:25.347370 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:25.347346 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:12:25.465403 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:25.465339 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" podStartSLOduration=2.619629258 podStartE2EDuration="3.465325198s" totalImagesPullingTime="845.69594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:25.462467908 +0000 UTC m=+1494.622540177" watchObservedRunningTime="2026-11-03 21:12:25.465325198 +0000 UTC m=+1494.625397469" Nov 03 21:12:25.952967 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:25.952930 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:27.168223 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:27.168111 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:29.427967 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:29.427924 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:31.130097 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:31.130062 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:12:31.138134 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:31.138104 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:12:31.358568 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:31.358532 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:12:31.549052 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:31.549022 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:31.734621 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:31.734584 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:31.917457 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:31.917359 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:35.395368 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:35.395330 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7481-ff3e00"] Nov 03 21:12:35.400927 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:35.400875 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-ff3e00" Nov 03 21:12:35.451960 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:35.451919 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-ff3e00"] Nov 03 21:12:35.492262 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:35.492219 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2mnp\" (UniqueName: \"kubernetes.io/projected/7cb3f554-d16a-469c-858a-4b4b1ad95060-kube-api-access-d2mnp\") pod \"test-curl-7481-ff3e00\" (UID: \"7cb3f554-d16a-469c-858a-4b4b1ad95060\") " pod="opendatahub/test-curl-7481-ff3e00" Nov 03 21:12:35.494298 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:35.494277 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7vxf5\"" Nov 03 21:12:35.593501 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:35.593459 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2mnp\" (UniqueName: \"kubernetes.io/projected/7cb3f554-d16a-469c-858a-4b4b1ad95060-kube-api-access-d2mnp\") pod \"test-curl-7481-ff3e00\" (UID: \"7cb3f554-d16a-469c-858a-4b4b1ad95060\") " pod="opendatahub/test-curl-7481-ff3e00" Nov 03 21:12:35.677295 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:35.677206 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2mnp\" (UniqueName: \"kubernetes.io/projected/7cb3f554-d16a-469c-858a-4b4b1ad95060-kube-api-access-d2mnp\") pod \"test-curl-7481-ff3e00\" (UID: \"7cb3f554-d16a-469c-858a-4b4b1ad95060\") " pod="opendatahub/test-curl-7481-ff3e00" Nov 03 21:12:35.712970 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:35.712937 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-ff3e00" Nov 03 21:12:36.086318 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:36.086277 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-ff3e00"] Nov 03 21:12:36.091326 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:12:36.091295 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cb3f554_d16a_469c_858a_4b4b1ad95060.slice/crio-43fe2b0ff5592891894d573941daa094eeb4925e58765da47b91418be3c1da0c WatchSource:0}: Error finding container 43fe2b0ff5592891894d573941daa094eeb4925e58765da47b91418be3c1da0c: Status 404 returned error can't find the container with id 43fe2b0ff5592891894d573941daa094eeb4925e58765da47b91418be3c1da0c Nov 03 21:12:36.323221 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:36.323188 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:36.464242 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:36.464203 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-ff3e00" event={"ID":"7cb3f554-d16a-469c-858a-4b4b1ad95060","Type":"ContainerStarted","Data":"43fe2b0ff5592891894d573941daa094eeb4925e58765da47b91418be3c1da0c"} Nov 03 21:12:38.481270 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:38.481229 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-ff3e00" event={"ID":"7cb3f554-d16a-469c-858a-4b4b1ad95060","Type":"ContainerStarted","Data":"bca40887cc50d9203950b9252f1b066dc804d7aee80c55d5e87677f26ca9ccc4"} Nov 03 21:12:38.592823 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:38.592745 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7481-ff3e00" podStartSLOduration=2.079363911 podStartE2EDuration="3.592725592s" totalImagesPullingTime="1.513361681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:38.589462149 +0000 UTC m=+1507.749534422" watchObservedRunningTime="2026-11-03 21:12:38.592725592 +0000 UTC m=+1507.752797867" Nov 03 21:12:40.129095 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:40.129042 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7481-ff3e00"] Nov 03 21:12:40.133375 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:40.133340 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7481-ff3e00"] Nov 03 21:12:40.494772 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:40.494736 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7481-ff3e00" podUID="7cb3f554-d16a-469c-858a-4b4b1ad95060" containerName="test-curl-7481-ff3e00" containerID="cri-o://bca40887cc50d9203950b9252f1b066dc804d7aee80c55d5e87677f26ca9ccc4" gracePeriod=2 Nov 03 21:12:40.543698 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:12:40.543661 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7481-ff3e00\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="7cb3f554-d16a-469c-858a-4b4b1ad95060" pod="opendatahub/test-curl-7481-ff3e00" Nov 03 21:12:40.890834 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:40.890736 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:41.014097 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:12:41.014053 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7481-ff3e00\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="7cb3f554-d16a-469c-858a-4b4b1ad95060" pod="opendatahub/test-curl-7481-ff3e00" Nov 03 21:12:42.529994 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:12:42.529890 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:12:42.749864 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:42.749789 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-ff3e00" Nov 03 21:12:42.801399 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:42.801361 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb3f554-d16a-469c-858a-4b4b1ad95060-kube-api-access-d2mnp\" (UniqueName: \"kubernetes.io/projected/7cb3f554-d16a-469c-858a-4b4b1ad95060-kube-api-access-d2mnp\") pod \"7cb3f554-d16a-469c-858a-4b4b1ad95060\" (UID: \"7cb3f554-d16a-469c-858a-4b4b1ad95060\") " Nov 03 21:12:42.804189 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:42.804155 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb3f554-d16a-469c-858a-4b4b1ad95060-kube-api-access-d2mnp" pod "7cb3f554-d16a-469c-858a-4b4b1ad95060" (UID: "7cb3f554-d16a-469c-858a-4b4b1ad95060"). InnerVolumeSpecName "kube-api-access-d2mnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:42.903361 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:42.903321 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2mnp\" (UniqueName: \"kubernetes.io/projected/7cb3f554-d16a-469c-858a-4b4b1ad95060-kube-api-access-d2mnp\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:42.969552 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:42.969514 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb3f554-d16a-469c-858a-4b4b1ad95060" path="/var/lib/kubelet/pods/7cb3f554-d16a-469c-858a-4b4b1ad95060/volumes" Nov 03 21:12:43.519787 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:43.519747 4963 generic.go:356] "Generic (PLEG): container finished" podID="7cb3f554-d16a-469c-858a-4b4b1ad95060" containerID="bca40887cc50d9203950b9252f1b066dc804d7aee80c55d5e87677f26ca9ccc4" exitCode=137 Nov 03 21:12:43.520011 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:43.519797 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-ff3e00" Nov 03 21:12:43.520011 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:43.519803 4963 scope.go:122] "RemoveContainer" containerID="bca40887cc50d9203950b9252f1b066dc804d7aee80c55d5e87677f26ca9ccc4" Nov 03 21:12:43.529390 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:43.529369 4963 scope.go:122] "RemoveContainer" containerID="bca40887cc50d9203950b9252f1b066dc804d7aee80c55d5e87677f26ca9ccc4" Nov 03 21:12:43.529700 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:12:43.529677 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca40887cc50d9203950b9252f1b066dc804d7aee80c55d5e87677f26ca9ccc4\": container with ID starting with bca40887cc50d9203950b9252f1b066dc804d7aee80c55d5e87677f26ca9ccc4 not found: ID does not exist" containerID="bca40887cc50d9203950b9252f1b066dc804d7aee80c55d5e87677f26ca9ccc4" Nov 03 21:12:43.529753 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:43.529714 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca40887cc50d9203950b9252f1b066dc804d7aee80c55d5e87677f26ca9ccc4"} err="failed to get container status \"bca40887cc50d9203950b9252f1b066dc804d7aee80c55d5e87677f26ca9ccc4\": rpc error: code = NotFound desc = could not find container \"bca40887cc50d9203950b9252f1b066dc804d7aee80c55d5e87677f26ca9ccc4\": container with ID starting with bca40887cc50d9203950b9252f1b066dc804d7aee80c55d5e87677f26ca9ccc4 not found: ID does not exist" Nov 03 21:12:45.777115 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:45.777072 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:50.535178 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:50.535142 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:53.206620 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:53.206580 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:53.410469 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:53.410433 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:55.102688 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:55.102655 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:59.621595 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:12:59.621558 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:04.269474 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:04.269438 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:08.574270 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:08.574233 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:09.045191 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:09.045152 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:12.529220 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:13:12.529183 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:13:13.799228 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:13.799182 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:14.300240 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:14.300202 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:17.973807 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:17.973771 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:18.157803 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:18.157769 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:18.747312 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:18.747273 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:20.166583 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:20.166553 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:23.397511 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:23.397479 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:25.912281 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:25.912234 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:26.094466 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:26.094431 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:26.640934 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:26.635168 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:27.072710 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:27.072679 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:28.216161 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:28.216125 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:29.776816 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:29.776775 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:31.405940 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:31.402396 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:31.616602 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:31.616511 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:33.253612 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:33.253576 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:34.088423 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:34.088384 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:37.610069 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:37.610028 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4"] Nov 03 21:13:37.610570 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:37.610287 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" podUID="84819654-ad13-4357-be29-5f8f17e05d1e" containerName="maas-api" containerID="cri-o://7e6f10cecf94bf8343656205ba7c2f961950c54a36b56cc7acb26a769eaeb63c" gracePeriod=30 Nov 03 21:13:37.702964 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:37.702887 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:37.893169 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:37.893075 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:37.919427 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:37.919393 4963 generic.go:356] "Generic (PLEG): container finished" podID="84819654-ad13-4357-be29-5f8f17e05d1e" containerID="7e6f10cecf94bf8343656205ba7c2f961950c54a36b56cc7acb26a769eaeb63c" exitCode=0 Nov 03 21:13:37.919568 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:37.919467 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" event={"ID":"84819654-ad13-4357-be29-5f8f17e05d1e","Type":"ContainerDied","Data":"7e6f10cecf94bf8343656205ba7c2f961950c54a36b56cc7acb26a769eaeb63c"} Nov 03 21:13:37.948816 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:37.948778 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:13:38.060123 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:38.060074 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84819654-ad13-4357-be29-5f8f17e05d1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84819654-ad13-4357-be29-5f8f17e05d1e-maas-api-tls\") pod \"84819654-ad13-4357-be29-5f8f17e05d1e\" (UID: \"84819654-ad13-4357-be29-5f8f17e05d1e\") " Nov 03 21:13:38.060329 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:38.060184 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84819654-ad13-4357-be29-5f8f17e05d1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84819654-ad13-4357-be29-5f8f17e05d1e-metrics-certs\") pod \"84819654-ad13-4357-be29-5f8f17e05d1e\" (UID: \"84819654-ad13-4357-be29-5f8f17e05d1e\") " Nov 03 21:13:38.060329 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:38.060280 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84819654-ad13-4357-be29-5f8f17e05d1e-kube-api-access-zzqqh\" (UniqueName: \"kubernetes.io/projected/84819654-ad13-4357-be29-5f8f17e05d1e-kube-api-access-zzqqh\") pod \"84819654-ad13-4357-be29-5f8f17e05d1e\" (UID: \"84819654-ad13-4357-be29-5f8f17e05d1e\") " Nov 03 21:13:38.060451 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:38.060329 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84819654-ad13-4357-be29-5f8f17e05d1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84819654-ad13-4357-be29-5f8f17e05d1e-openshift-service-ca\") pod \"84819654-ad13-4357-be29-5f8f17e05d1e\" (UID: \"84819654-ad13-4357-be29-5f8f17e05d1e\") " Nov 03 21:13:38.061703 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:38.061587 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84819654-ad13-4357-be29-5f8f17e05d1e-openshift-service-ca" pod "84819654-ad13-4357-be29-5f8f17e05d1e" (UID: "84819654-ad13-4357-be29-5f8f17e05d1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:38.064202 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:38.064173 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84819654-ad13-4357-be29-5f8f17e05d1e-metrics-certs" pod "84819654-ad13-4357-be29-5f8f17e05d1e" (UID: "84819654-ad13-4357-be29-5f8f17e05d1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:38.064440 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:38.064382 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84819654-ad13-4357-be29-5f8f17e05d1e-maas-api-tls" pod "84819654-ad13-4357-be29-5f8f17e05d1e" (UID: "84819654-ad13-4357-be29-5f8f17e05d1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:38.064774 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:38.064744 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84819654-ad13-4357-be29-5f8f17e05d1e-kube-api-access-zzqqh" pod "84819654-ad13-4357-be29-5f8f17e05d1e" (UID: "84819654-ad13-4357-be29-5f8f17e05d1e"). InnerVolumeSpecName "kube-api-access-zzqqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:38.162533 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:38.162433 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzqqh\" (UniqueName: \"kubernetes.io/projected/84819654-ad13-4357-be29-5f8f17e05d1e-kube-api-access-zzqqh\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:38.162533 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:38.162476 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84819654-ad13-4357-be29-5f8f17e05d1e-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:38.162533 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:38.162493 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84819654-ad13-4357-be29-5f8f17e05d1e-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:38.162533 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:38.162503 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84819654-ad13-4357-be29-5f8f17e05d1e-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:38.412967 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:38.412854 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:38.932671 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:38.932641 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" Nov 03 21:13:38.932671 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:38.932651 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4" event={"ID":"84819654-ad13-4357-be29-5f8f17e05d1e","Type":"ContainerDied","Data":"07139ae99bdb7711471d04784bef86df660b7bbe56fb294d05249df42c5108af"} Nov 03 21:13:38.933225 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:38.932720 4963 scope.go:122] "RemoveContainer" containerID="7e6f10cecf94bf8343656205ba7c2f961950c54a36b56cc7acb26a769eaeb63c" Nov 03 21:13:39.074057 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:39.073992 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:39.182883 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:39.182790 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4"] Nov 03 21:13:39.228623 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:39.228587 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-145cc5-76df675754-gq4v4"] Nov 03 21:13:39.297966 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:39.297932 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:39.493659 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:39.493620 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:39.707952 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:39.705760 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:40.971813 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:40.971595 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84819654-ad13-4357-be29-5f8f17e05d1e" path="/var/lib/kubelet/pods/84819654-ad13-4357-be29-5f8f17e05d1e/volumes" Nov 03 21:13:41.191283 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:41.191236 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:41.377194 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:41.377114 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:42.528485 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:13:42.528443 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:13:42.613828 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:42.613793 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:43.113134 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:43.113088 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:43.926158 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:43.926125 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:44.107027 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:44.106991 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:44.610395 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:44.610362 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:44.799993 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:44.799868 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:44.987696 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:44.987667 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:45.179406 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:45.179368 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:46.763777 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:46.763746 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:48.476108 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:48.476075 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:48.661137 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:48.661103 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:49.703678 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:49.703649 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:50.656173 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:50.656140 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:50.857783 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:50.857752 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:52.514877 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:52.514843 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:52.700848 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:52.700810 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:54.503768 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:54.503728 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:56.751664 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:56.751622 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:58.751234 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:13:58.751198 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:00.509432 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:00.509384 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:05.155192 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:05.155085 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:06.277672 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:06.277634 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:06.466158 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:06.466128 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:07.704676 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:07.704642 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:08.251804 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:08.251764 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:09.085841 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:09.085798 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:09.943947 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:09.943898 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:10.125691 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:10.125659 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:12.529373 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:14:12.529334 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:14:14.834176 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:14.834141 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:15.272164 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:15.272132 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:16.457887 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:16.457856 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:16.644599 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:16.644566 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:19.387658 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:19.387620 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:19.608400 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:19.608365 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:19.790594 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:19.790556 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:21.192226 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:21.192186 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:24.097144 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:24.097105 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:24.805419 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:24.805384 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:24.990138 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:24.990102 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:28.066105 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:28.066062 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:29.489833 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:29.489799 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:29.708197 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:29.708164 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:29.896313 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:29.896232 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:30.117402 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:30.117365 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:34.088310 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:34.088273 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:34.498688 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:34.498644 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:34.681392 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:34.681362 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:34.864066 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:34.863983 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:37.767525 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:37.767488 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:37.949780 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:37.949734 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:38.397334 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:38.397300 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:39.542020 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:39.541988 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:42.170401 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:42.170365 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:42.361499 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:42.361467 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:42.554672 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:14:42.554633 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:14:43.695959 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:43.695925 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg"] Nov 03 21:14:43.696402 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:43.696135 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" podUID="bab7cfd1-1ae1-4858-8b51-ed89214a47de" containerName="maas-api" containerID="cri-o://0cc0c4b606292ba91263b54ce67f9a104bb10b042e6c9d6ea0bc79a110c0d18c" gracePeriod=30 Nov 03 21:14:43.835424 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:43.835387 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:44.032530 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.032496 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:14:44.189310 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.189274 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bab7cfd1-1ae1-4858-8b51-ed89214a47de-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab7cfd1-1ae1-4858-8b51-ed89214a47de-openshift-service-ca\") pod \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\" (UID: \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\") " Nov 03 21:14:44.189509 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.189403 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab7cfd1-1ae1-4858-8b51-ed89214a47de-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab7cfd1-1ae1-4858-8b51-ed89214a47de-maas-api-tls\") pod \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\" (UID: \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\") " Nov 03 21:14:44.189593 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.189540 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab7cfd1-1ae1-4858-8b51-ed89214a47de-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab7cfd1-1ae1-4858-8b51-ed89214a47de-metrics-certs\") pod \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\" (UID: \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\") " Nov 03 21:14:44.189681 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.189648 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab7cfd1-1ae1-4858-8b51-ed89214a47de-kube-api-access-lm6r9\" (UniqueName: \"kubernetes.io/projected/bab7cfd1-1ae1-4858-8b51-ed89214a47de-kube-api-access-lm6r9\") pod \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\" (UID: \"bab7cfd1-1ae1-4858-8b51-ed89214a47de\") " Nov 03 21:14:44.189740 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.189687 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab7cfd1-1ae1-4858-8b51-ed89214a47de-openshift-service-ca" pod "bab7cfd1-1ae1-4858-8b51-ed89214a47de" (UID: "bab7cfd1-1ae1-4858-8b51-ed89214a47de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:44.190288 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.190265 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab7cfd1-1ae1-4858-8b51-ed89214a47de-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:44.192039 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.192015 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab7cfd1-1ae1-4858-8b51-ed89214a47de-metrics-certs" pod "bab7cfd1-1ae1-4858-8b51-ed89214a47de" (UID: "bab7cfd1-1ae1-4858-8b51-ed89214a47de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:14:44.192245 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.192229 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab7cfd1-1ae1-4858-8b51-ed89214a47de-maas-api-tls" pod "bab7cfd1-1ae1-4858-8b51-ed89214a47de" (UID: "bab7cfd1-1ae1-4858-8b51-ed89214a47de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:14:44.192315 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.192231 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab7cfd1-1ae1-4858-8b51-ed89214a47de-kube-api-access-lm6r9" pod "bab7cfd1-1ae1-4858-8b51-ed89214a47de" (UID: "bab7cfd1-1ae1-4858-8b51-ed89214a47de"). InnerVolumeSpecName "kube-api-access-lm6r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:44.293019 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.292886 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab7cfd1-1ae1-4858-8b51-ed89214a47de-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:44.293019 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.292948 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab7cfd1-1ae1-4858-8b51-ed89214a47de-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:44.293019 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.292992 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm6r9\" (UniqueName: \"kubernetes.io/projected/bab7cfd1-1ae1-4858-8b51-ed89214a47de-kube-api-access-lm6r9\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:44.470088 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.470051 4963 generic.go:356] "Generic (PLEG): container finished" podID="bab7cfd1-1ae1-4858-8b51-ed89214a47de" containerID="0cc0c4b606292ba91263b54ce67f9a104bb10b042e6c9d6ea0bc79a110c0d18c" exitCode=0 Nov 03 21:14:44.470284 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.470099 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" Nov 03 21:14:44.470284 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.470139 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" event={"ID":"bab7cfd1-1ae1-4858-8b51-ed89214a47de","Type":"ContainerDied","Data":"0cc0c4b606292ba91263b54ce67f9a104bb10b042e6c9d6ea0bc79a110c0d18c"} Nov 03 21:14:44.470284 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.470226 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg" event={"ID":"bab7cfd1-1ae1-4858-8b51-ed89214a47de","Type":"ContainerDied","Data":"30d04f54bdc530d89a68dae64e28d320dc8c5bed3334ed27d7f8542f0f780015"} Nov 03 21:14:44.470284 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.470250 4963 scope.go:122] "RemoveContainer" containerID="0cc0c4b606292ba91263b54ce67f9a104bb10b042e6c9d6ea0bc79a110c0d18c" Nov 03 21:14:44.481075 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.481052 4963 scope.go:122] "RemoveContainer" containerID="0cc0c4b606292ba91263b54ce67f9a104bb10b042e6c9d6ea0bc79a110c0d18c" Nov 03 21:14:44.481397 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:14:44.481378 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cc0c4b606292ba91263b54ce67f9a104bb10b042e6c9d6ea0bc79a110c0d18c\": container with ID starting with 0cc0c4b606292ba91263b54ce67f9a104bb10b042e6c9d6ea0bc79a110c0d18c not found: ID does not exist" containerID="0cc0c4b606292ba91263b54ce67f9a104bb10b042e6c9d6ea0bc79a110c0d18c" Nov 03 21:14:44.481467 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.481417 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc0c4b606292ba91263b54ce67f9a104bb10b042e6c9d6ea0bc79a110c0d18c"} err="failed to get container status \"0cc0c4b606292ba91263b54ce67f9a104bb10b042e6c9d6ea0bc79a110c0d18c\": rpc error: code = NotFound desc = could not find container \"0cc0c4b606292ba91263b54ce67f9a104bb10b042e6c9d6ea0bc79a110c0d18c\": container with ID starting with 0cc0c4b606292ba91263b54ce67f9a104bb10b042e6c9d6ea0bc79a110c0d18c not found: ID does not exist" Nov 03 21:14:44.628066 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.627963 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg"] Nov 03 21:14:44.636723 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.636691 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6ad223-569f4d679b-wbspg"] Nov 03 21:14:44.968458 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:44.968417 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab7cfd1-1ae1-4858-8b51-ed89214a47de" path="/var/lib/kubelet/pods/bab7cfd1-1ae1-4858-8b51-ed89214a47de/volumes" Nov 03 21:14:45.657421 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:45.657325 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:48.124502 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:48.124462 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:48.312695 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:48.312662 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:49.660500 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:49.660470 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:50.153782 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:50.153745 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:50.640384 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:50.640347 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:51.931736 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:51.931701 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:52.939393 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:52.939359 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:53.254627 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:53.254593 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:55.494195 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:55.494161 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:57.634001 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:57.633966 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:57.936010 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:57.935892 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:59.685205 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:14:59.685170 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:00.165302 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.165267 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm"] Nov 03 21:15:00.165949 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.165928 4963 cpu_manager.go:414] "containerMap: removing container" podUID="84819654-ad13-4357-be29-5f8f17e05d1e" containerName="maas-api" Nov 03 21:15:00.165949 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.165948 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84819654-ad13-4357-be29-5f8f17e05d1e" containerName="maas-api" Nov 03 21:15:00.166132 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.165992 4963 cpu_manager.go:414] "containerMap: removing container" podUID="7cb3f554-d16a-469c-858a-4b4b1ad95060" containerName="test-curl-7481-ff3e00" Nov 03 21:15:00.166132 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.166000 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb3f554-d16a-469c-858a-4b4b1ad95060" containerName="test-curl-7481-ff3e00" Nov 03 21:15:00.166132 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.166023 4963 cpu_manager.go:414] "containerMap: removing container" podUID="bab7cfd1-1ae1-4858-8b51-ed89214a47de" containerName="maas-api" Nov 03 21:15:00.166132 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.166030 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab7cfd1-1ae1-4858-8b51-ed89214a47de" containerName="maas-api" Nov 03 21:15:00.166322 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.166197 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab7cfd1-1ae1-4858-8b51-ed89214a47de" containerName="maas-api" Nov 03 21:15:00.166322 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.166215 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="84819654-ad13-4357-be29-5f8f17e05d1e" containerName="maas-api" Nov 03 21:15:00.166322 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.166231 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb3f554-d16a-469c-858a-4b4b1ad95060" containerName="test-curl-7481-ff3e00" Nov 03 21:15:00.169751 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.169729 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" Nov 03 21:15:00.183981 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.183954 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz"] Nov 03 21:15:00.188300 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.188280 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" Nov 03 21:15:00.189389 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.189367 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d"] Nov 03 21:15:00.193357 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.193338 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" Nov 03 21:15:00.227996 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.227955 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\"" Nov 03 21:15:00.259107 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.259059 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm"] Nov 03 21:15:00.309995 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.309953 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4386438d-d439-49f4-8dcb-b339637c624e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm\" (UID: \"4386438d-d439-49f4-8dcb-b339637c624e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" Nov 03 21:15:00.310187 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.310043 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv4lm\" (UniqueName: \"kubernetes.io/projected/4386438d-d439-49f4-8dcb-b339637c624e-kube-api-access-wv4lm\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm\" (UID: \"4386438d-d439-49f4-8dcb-b339637c624e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" Nov 03 21:15:00.310262 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.310189 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9eb3d46-e4da-4ec9-9a92-0b78a84796b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d\" (UID: \"a9eb3d46-e4da-4ec9-9a92-0b78a84796b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" Nov 03 21:15:00.310262 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.310226 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj62j\" (UniqueName: \"kubernetes.io/projected/a9eb3d46-e4da-4ec9-9a92-0b78a84796b6-kube-api-access-wj62j\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d\" (UID: \"a9eb3d46-e4da-4ec9-9a92-0b78a84796b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" Nov 03 21:15:00.310262 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.310258 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzrw5\" (UniqueName: \"kubernetes.io/projected/cf1ffc34-0f1d-4022-bb27-bea026dfd5fd-kube-api-access-qzrw5\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz\" (UID: \"cf1ffc34-0f1d-4022-bb27-bea026dfd5fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" Nov 03 21:15:00.310384 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.310287 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf1ffc34-0f1d-4022-bb27-bea026dfd5fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz\" (UID: \"cf1ffc34-0f1d-4022-bb27-bea026dfd5fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" Nov 03 21:15:00.310479 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.310459 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz"] Nov 03 21:15:00.315465 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.315432 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d"] Nov 03 21:15:00.406765 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.406697 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:00.411592 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.411561 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4386438d-d439-49f4-8dcb-b339637c624e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm\" (UID: \"4386438d-d439-49f4-8dcb-b339637c624e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" Nov 03 21:15:00.411790 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.411626 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv4lm\" (UniqueName: \"kubernetes.io/projected/4386438d-d439-49f4-8dcb-b339637c624e-kube-api-access-wv4lm\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm\" (UID: \"4386438d-d439-49f4-8dcb-b339637c624e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" Nov 03 21:15:00.412464 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.412438 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4386438d-d439-49f4-8dcb-b339637c624e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm\" (UID: \"4386438d-d439-49f4-8dcb-b339637c624e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" Nov 03 21:15:00.412609 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.412586 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9eb3d46-e4da-4ec9-9a92-0b78a84796b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d\" (UID: \"a9eb3d46-e4da-4ec9-9a92-0b78a84796b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" Nov 03 21:15:00.412700 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.411769 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9eb3d46-e4da-4ec9-9a92-0b78a84796b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d\" (UID: \"a9eb3d46-e4da-4ec9-9a92-0b78a84796b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" Nov 03 21:15:00.412750 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.412736 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj62j\" (UniqueName: \"kubernetes.io/projected/a9eb3d46-e4da-4ec9-9a92-0b78a84796b6-kube-api-access-wj62j\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d\" (UID: \"a9eb3d46-e4da-4ec9-9a92-0b78a84796b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" Nov 03 21:15:00.412785 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.412763 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrw5\" (UniqueName: \"kubernetes.io/projected/cf1ffc34-0f1d-4022-bb27-bea026dfd5fd-kube-api-access-qzrw5\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz\" (UID: \"cf1ffc34-0f1d-4022-bb27-bea026dfd5fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" Nov 03 21:15:00.412819 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.412787 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf1ffc34-0f1d-4022-bb27-bea026dfd5fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz\" (UID: \"cf1ffc34-0f1d-4022-bb27-bea026dfd5fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" Nov 03 21:15:00.413513 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.413492 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf1ffc34-0f1d-4022-bb27-bea026dfd5fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz\" (UID: \"cf1ffc34-0f1d-4022-bb27-bea026dfd5fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" Nov 03 21:15:00.471071 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.471011 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv4lm\" (UniqueName: \"kubernetes.io/projected/4386438d-d439-49f4-8dcb-b339637c624e-kube-api-access-wv4lm\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm\" (UID: \"4386438d-d439-49f4-8dcb-b339637c624e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" Nov 03 21:15:00.471868 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.471841 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj62j\" (UniqueName: \"kubernetes.io/projected/a9eb3d46-e4da-4ec9-9a92-0b78a84796b6-kube-api-access-wj62j\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d\" (UID: \"a9eb3d46-e4da-4ec9-9a92-0b78a84796b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" Nov 03 21:15:00.474583 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.474556 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzrw5\" (UniqueName: \"kubernetes.io/projected/cf1ffc34-0f1d-4022-bb27-bea026dfd5fd-kube-api-access-qzrw5\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz\" (UID: \"cf1ffc34-0f1d-4022-bb27-bea026dfd5fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" Nov 03 21:15:00.483128 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.483102 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" Nov 03 21:15:00.505178 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.505147 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" Nov 03 21:15:00.509545 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.509521 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" Nov 03 21:15:00.879796 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.879704 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm"] Nov 03 21:15:00.884324 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:15:00.884298 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4386438d_d439_49f4_8dcb_b339637c624e.slice/crio-7b0b11c3e2b37ff5de8d37db80a1bdeec0091dcde0d6fd7063223fd542f8a246 WatchSource:0}: Error finding container 7b0b11c3e2b37ff5de8d37db80a1bdeec0091dcde0d6fd7063223fd542f8a246: Status 404 returned error can't find the container with id 7b0b11c3e2b37ff5de8d37db80a1bdeec0091dcde0d6fd7063223fd542f8a246 Nov 03 21:15:00.930366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.930331 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d"] Nov 03 21:15:00.934165 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:00.934137 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz"] Nov 03 21:15:00.935682 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:15:00.935656 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9eb3d46_e4da_4ec9_9a92_0b78a84796b6.slice/crio-5a14915556871b8a8bee7e726c9d01964dee3262e5610ee8dcce155ad22ee427 WatchSource:0}: Error finding container 5a14915556871b8a8bee7e726c9d01964dee3262e5610ee8dcce155ad22ee427: Status 404 returned error can't find the container with id 5a14915556871b8a8bee7e726c9d01964dee3262e5610ee8dcce155ad22ee427 Nov 03 21:15:00.937981 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:15:00.937954 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf1ffc34_0f1d_4022_bb27_bea026dfd5fd.slice/crio-e2a0e779b357930d286e7ef72af8c1047d9e56de99296437f65fc382864a1b0f WatchSource:0}: Error finding container e2a0e779b357930d286e7ef72af8c1047d9e56de99296437f65fc382864a1b0f: Status 404 returned error can't find the container with id e2a0e779b357930d286e7ef72af8c1047d9e56de99296437f65fc382864a1b0f Nov 03 21:15:01.596581 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:01.596469 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" event={"ID":"a9eb3d46-e4da-4ec9-9a92-0b78a84796b6","Type":"ContainerStarted","Data":"9663b98a948eb1ed9a2a688236c198f3adeb9fefbaeed85179aa6ab088e13107"} Nov 03 21:15:01.596581 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:01.596531 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" event={"ID":"a9eb3d46-e4da-4ec9-9a92-0b78a84796b6","Type":"ContainerStarted","Data":"5a14915556871b8a8bee7e726c9d01964dee3262e5610ee8dcce155ad22ee427"} Nov 03 21:15:01.598540 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:01.598367 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" event={"ID":"cf1ffc34-0f1d-4022-bb27-bea026dfd5fd","Type":"ContainerStarted","Data":"890272800ebab1f5aabf20659fba309608747de1ac9effc6d581a03e0ef90d70"} Nov 03 21:15:01.598540 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:01.598443 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" event={"ID":"cf1ffc34-0f1d-4022-bb27-bea026dfd5fd","Type":"ContainerStarted","Data":"e2a0e779b357930d286e7ef72af8c1047d9e56de99296437f65fc382864a1b0f"} Nov 03 21:15:01.600338 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:01.600313 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" event={"ID":"4386438d-d439-49f4-8dcb-b339637c624e","Type":"ContainerStarted","Data":"da7fd244e783f1cf0ecfd09137566ba4c2030053436b23db9e64681746293700"} Nov 03 21:15:01.600479 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:01.600359 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" event={"ID":"4386438d-d439-49f4-8dcb-b339637c624e","Type":"ContainerStarted","Data":"7b0b11c3e2b37ff5de8d37db80a1bdeec0091dcde0d6fd7063223fd542f8a246"} Nov 03 21:15:01.708716 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:01.708659 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" podStartSLOduration=1.708644305 podStartE2EDuration="1.708644305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:01.70584694 +0000 UTC m=+1650.865919213" watchObservedRunningTime="2026-11-03 21:15:01.708644305 +0000 UTC m=+1650.868716576" Nov 03 21:15:01.817480 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:01.817431 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" podStartSLOduration=1.817406263 podStartE2EDuration="1.817406263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:01.814961135 +0000 UTC m=+1650.975033408" watchObservedRunningTime="2026-11-03 21:15:01.817406263 +0000 UTC m=+1650.977478555" Nov 03 21:15:01.924963 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:01.924863 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" podStartSLOduration=1.924848546 podStartE2EDuration="1.924848546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:01.923748846 +0000 UTC m=+1651.083821117" watchObservedRunningTime="2026-11-03 21:15:01.924848546 +0000 UTC m=+1651.084920846" Nov 03 21:15:02.189150 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:02.189061 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:02.381621 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:02.381589 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:02.643560 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:02.642522 4963 generic.go:356] "Generic (PLEG): container finished" podID="a9eb3d46-e4da-4ec9-9a92-0b78a84796b6" containerID="9663b98a948eb1ed9a2a688236c198f3adeb9fefbaeed85179aa6ab088e13107" exitCode=60 Nov 03 21:15:02.643560 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:02.643506 4963 scope.go:122] "RemoveContainer" containerID="9663b98a948eb1ed9a2a688236c198f3adeb9fefbaeed85179aa6ab088e13107" Nov 03 21:15:02.643993 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:02.643963 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" event={"ID":"a9eb3d46-e4da-4ec9-9a92-0b78a84796b6","Type":"ContainerDied","Data":"9663b98a948eb1ed9a2a688236c198f3adeb9fefbaeed85179aa6ab088e13107"} Nov 03 21:15:02.647860 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:02.647832 4963 generic.go:356] "Generic (PLEG): container finished" podID="cf1ffc34-0f1d-4022-bb27-bea026dfd5fd" containerID="890272800ebab1f5aabf20659fba309608747de1ac9effc6d581a03e0ef90d70" exitCode=60 Nov 03 21:15:02.648172 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:02.648123 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" event={"ID":"cf1ffc34-0f1d-4022-bb27-bea026dfd5fd","Type":"ContainerDied","Data":"890272800ebab1f5aabf20659fba309608747de1ac9effc6d581a03e0ef90d70"} Nov 03 21:15:02.649186 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:02.649167 4963 scope.go:122] "RemoveContainer" containerID="890272800ebab1f5aabf20659fba309608747de1ac9effc6d581a03e0ef90d70" Nov 03 21:15:02.653204 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:02.653149 4963 generic.go:356] "Generic (PLEG): container finished" podID="4386438d-d439-49f4-8dcb-b339637c624e" containerID="da7fd244e783f1cf0ecfd09137566ba4c2030053436b23db9e64681746293700" exitCode=60 Nov 03 21:15:02.653314 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:02.653272 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" event={"ID":"4386438d-d439-49f4-8dcb-b339637c624e","Type":"ContainerDied","Data":"da7fd244e783f1cf0ecfd09137566ba4c2030053436b23db9e64681746293700"} Nov 03 21:15:02.653729 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:02.653705 4963 scope.go:122] "RemoveContainer" containerID="da7fd244e783f1cf0ecfd09137566ba4c2030053436b23db9e64681746293700" Nov 03 21:15:03.662794 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:03.662758 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" event={"ID":"a9eb3d46-e4da-4ec9-9a92-0b78a84796b6","Type":"ContainerStarted","Data":"2e11c85e4c8c5ea290b9984884265be0a40b9f42c3f9b033507ad1f030175acb"} Nov 03 21:15:03.664732 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:03.664706 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" event={"ID":"cf1ffc34-0f1d-4022-bb27-bea026dfd5fd","Type":"ContainerStarted","Data":"b8201dbed2f88e6d0c27631ccda1c57c6e3a38df087725e0301e7edb859faeea"} Nov 03 21:15:03.666563 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:03.666534 4963 generic.go:356] "Generic (PLEG): container finished" podID="4386438d-d439-49f4-8dcb-b339637c624e" containerID="84c17d2f86245e66958b0e70c8295a18c89878afa5a4f8f150a6edf7f573696f" exitCode=60 Nov 03 21:15:03.666685 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:03.666570 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" event={"ID":"4386438d-d439-49f4-8dcb-b339637c624e","Type":"ContainerDied","Data":"84c17d2f86245e66958b0e70c8295a18c89878afa5a4f8f150a6edf7f573696f"} Nov 03 21:15:03.666685 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:03.666628 4963 scope.go:122] "RemoveContainer" containerID="da7fd244e783f1cf0ecfd09137566ba4c2030053436b23db9e64681746293700" Nov 03 21:15:03.666958 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:03.666937 4963 scope.go:122] "RemoveContainer" containerID="84c17d2f86245e66958b0e70c8295a18c89878afa5a4f8f150a6edf7f573696f" Nov 03 21:15:03.667233 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:15:03.667207 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm_odh-ai-gateway-infra(4386438d-d439-49f4-8dcb-b339637c624e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" podUID="4386438d-d439-49f4-8dcb-b339637c624e" Nov 03 21:15:04.642226 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:04.642187 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:04.677864 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:04.677827 4963 generic.go:356] "Generic (PLEG): container finished" podID="a9eb3d46-e4da-4ec9-9a92-0b78a84796b6" containerID="2e11c85e4c8c5ea290b9984884265be0a40b9f42c3f9b033507ad1f030175acb" exitCode=60 Nov 03 21:15:04.678392 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:04.677944 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" event={"ID":"a9eb3d46-e4da-4ec9-9a92-0b78a84796b6","Type":"ContainerDied","Data":"2e11c85e4c8c5ea290b9984884265be0a40b9f42c3f9b033507ad1f030175acb"} Nov 03 21:15:04.678392 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:04.678016 4963 scope.go:122] "RemoveContainer" containerID="9663b98a948eb1ed9a2a688236c198f3adeb9fefbaeed85179aa6ab088e13107" Nov 03 21:15:04.678392 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:04.678363 4963 scope.go:122] "RemoveContainer" containerID="2e11c85e4c8c5ea290b9984884265be0a40b9f42c3f9b033507ad1f030175acb" Nov 03 21:15:04.678762 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:15:04.678736 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d_odh-ai-gateway-infra(a9eb3d46-e4da-4ec9-9a92-0b78a84796b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" podUID="a9eb3d46-e4da-4ec9-9a92-0b78a84796b6" Nov 03 21:15:04.681898 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:04.681870 4963 generic.go:356] "Generic (PLEG): container finished" podID="cf1ffc34-0f1d-4022-bb27-bea026dfd5fd" containerID="b8201dbed2f88e6d0c27631ccda1c57c6e3a38df087725e0301e7edb859faeea" exitCode=60 Nov 03 21:15:04.682050 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:04.681971 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" event={"ID":"cf1ffc34-0f1d-4022-bb27-bea026dfd5fd","Type":"ContainerDied","Data":"b8201dbed2f88e6d0c27631ccda1c57c6e3a38df087725e0301e7edb859faeea"} Nov 03 21:15:04.682302 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:04.682287 4963 scope.go:122] "RemoveContainer" containerID="b8201dbed2f88e6d0c27631ccda1c57c6e3a38df087725e0301e7edb859faeea" Nov 03 21:15:04.682595 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:15:04.682571 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz_odh-ai-gateway-infra(cf1ffc34-0f1d-4022-bb27-bea026dfd5fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" podUID="cf1ffc34-0f1d-4022-bb27-bea026dfd5fd" Nov 03 21:15:04.684307 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:04.684287 4963 scope.go:122] "RemoveContainer" containerID="84c17d2f86245e66958b0e70c8295a18c89878afa5a4f8f150a6edf7f573696f" Nov 03 21:15:04.684609 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:15:04.684588 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm_odh-ai-gateway-infra(4386438d-d439-49f4-8dcb-b339637c624e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" podUID="4386438d-d439-49f4-8dcb-b339637c624e" Nov 03 21:15:04.695614 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:04.695579 4963 scope.go:122] "RemoveContainer" containerID="890272800ebab1f5aabf20659fba309608747de1ac9effc6d581a03e0ef90d70" Nov 03 21:15:05.692428 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:05.692394 4963 scope.go:122] "RemoveContainer" containerID="2e11c85e4c8c5ea290b9984884265be0a40b9f42c3f9b033507ad1f030175acb" Nov 03 21:15:05.692892 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:15:05.692752 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d_odh-ai-gateway-infra(a9eb3d46-e4da-4ec9-9a92-0b78a84796b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" podUID="a9eb3d46-e4da-4ec9-9a92-0b78a84796b6" Nov 03 21:15:05.694118 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:05.694097 4963 scope.go:122] "RemoveContainer" containerID="b8201dbed2f88e6d0c27631ccda1c57c6e3a38df087725e0301e7edb859faeea" Nov 03 21:15:05.694427 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:15:05.694405 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz_odh-ai-gateway-infra(cf1ffc34-0f1d-4022-bb27-bea026dfd5fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" podUID="cf1ffc34-0f1d-4022-bb27-bea026dfd5fd" Nov 03 21:15:06.423496 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:06.422069 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:11.308197 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:11.308057 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:12.336035 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:12.335988 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:12.515044 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:12.515004 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:12.529834 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:15:12.529800 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:15:13.960863 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:13.960835 4963 scope.go:122] "RemoveContainer" containerID="84c17d2f86245e66958b0e70c8295a18c89878afa5a4f8f150a6edf7f573696f" Nov 03 21:15:14.077898 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:14.077854 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:14.567826 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:14.567742 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:14.754480 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:14.754437 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:14.758324 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:14.758297 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" event={"ID":"4386438d-d439-49f4-8dcb-b339637c624e","Type":"ContainerStarted","Data":"6350862b1c02f5382748ca90eacdae2c756988dbe285943401aeeab7ae5858cb"} Nov 03 21:15:14.960689 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:14.960661 4963 scope.go:122] "RemoveContainer" containerID="b8201dbed2f88e6d0c27631ccda1c57c6e3a38df087725e0301e7edb859faeea" Nov 03 21:15:15.250137 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:15.250093 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:15.766580 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:15.766545 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" event={"ID":"cf1ffc34-0f1d-4022-bb27-bea026dfd5fd","Type":"ContainerStarted","Data":"d2f9a06adaddd909aa9952deb6f9b96a2c7dd684019d74017c46bf6cdf5cbe9b"} Nov 03 21:15:15.768518 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:15.768493 4963 generic.go:356] "Generic (PLEG): container finished" podID="4386438d-d439-49f4-8dcb-b339637c624e" containerID="6350862b1c02f5382748ca90eacdae2c756988dbe285943401aeeab7ae5858cb" exitCode=60 Nov 03 21:15:15.768645 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:15.768572 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" event={"ID":"4386438d-d439-49f4-8dcb-b339637c624e","Type":"ContainerDied","Data":"6350862b1c02f5382748ca90eacdae2c756988dbe285943401aeeab7ae5858cb"} Nov 03 21:15:15.768645 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:15.768633 4963 scope.go:122] "RemoveContainer" containerID="84c17d2f86245e66958b0e70c8295a18c89878afa5a4f8f150a6edf7f573696f" Nov 03 21:15:15.768959 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:15.768945 4963 scope.go:122] "RemoveContainer" containerID="6350862b1c02f5382748ca90eacdae2c756988dbe285943401aeeab7ae5858cb" Nov 03 21:15:15.769254 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:15:15.769232 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm_odh-ai-gateway-infra(4386438d-d439-49f4-8dcb-b339637c624e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" podUID="4386438d-d439-49f4-8dcb-b339637c624e" Nov 03 21:15:15.879857 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:15.879824 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm"] Nov 03 21:15:16.054858 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:16.054781 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:16.241306 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:16.241274 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:16.428594 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:16.428481 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:16.765329 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:16.765163 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:16.798279 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:16.798244 4963 generic.go:356] "Generic (PLEG): container finished" podID="cf1ffc34-0f1d-4022-bb27-bea026dfd5fd" containerID="d2f9a06adaddd909aa9952deb6f9b96a2c7dd684019d74017c46bf6cdf5cbe9b" exitCode=60 Nov 03 21:15:16.798574 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:16.798546 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" event={"ID":"cf1ffc34-0f1d-4022-bb27-bea026dfd5fd","Type":"ContainerDied","Data":"d2f9a06adaddd909aa9952deb6f9b96a2c7dd684019d74017c46bf6cdf5cbe9b"} Nov 03 21:15:16.798820 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:16.798802 4963 scope.go:122] "RemoveContainer" containerID="b8201dbed2f88e6d0c27631ccda1c57c6e3a38df087725e0301e7edb859faeea" Nov 03 21:15:16.799419 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:16.799400 4963 scope.go:122] "RemoveContainer" containerID="d2f9a06adaddd909aa9952deb6f9b96a2c7dd684019d74017c46bf6cdf5cbe9b" Nov 03 21:15:16.799965 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:15:16.799939 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz_odh-ai-gateway-infra(cf1ffc34-0f1d-4022-bb27-bea026dfd5fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" podUID="cf1ffc34-0f1d-4022-bb27-bea026dfd5fd" Nov 03 21:15:16.905147 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:16.904928 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz"] Nov 03 21:15:16.965932 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:16.965029 4963 scope.go:122] "RemoveContainer" containerID="2e11c85e4c8c5ea290b9984884265be0a40b9f42c3f9b033507ad1f030175acb" Nov 03 21:15:17.049352 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:17.038149 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:17.098387 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:17.098355 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" Nov 03 21:15:17.188306 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:17.188275 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4386438d-d439-49f4-8dcb-b339637c624e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4386438d-d439-49f4-8dcb-b339637c624e-maas-api-service-ca\") pod \"4386438d-d439-49f4-8dcb-b339637c624e\" (UID: \"4386438d-d439-49f4-8dcb-b339637c624e\") " Nov 03 21:15:17.188472 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:17.188429 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4386438d-d439-49f4-8dcb-b339637c624e-kube-api-access-wv4lm\" (UniqueName: \"kubernetes.io/projected/4386438d-d439-49f4-8dcb-b339637c624e-kube-api-access-wv4lm\") pod \"4386438d-d439-49f4-8dcb-b339637c624e\" (UID: \"4386438d-d439-49f4-8dcb-b339637c624e\") " Nov 03 21:15:17.188613 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:17.188592 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4386438d-d439-49f4-8dcb-b339637c624e-maas-api-service-ca" pod "4386438d-d439-49f4-8dcb-b339637c624e" (UID: "4386438d-d439-49f4-8dcb-b339637c624e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:17.189070 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:17.189049 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4386438d-d439-49f4-8dcb-b339637c624e-maas-api-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:17.190642 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:17.190624 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4386438d-d439-49f4-8dcb-b339637c624e-kube-api-access-wv4lm" pod "4386438d-d439-49f4-8dcb-b339637c624e" (UID: "4386438d-d439-49f4-8dcb-b339637c624e"). InnerVolumeSpecName "kube-api-access-wv4lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:17.290261 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:17.290227 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv4lm\" (UniqueName: \"kubernetes.io/projected/4386438d-d439-49f4-8dcb-b339637c624e-kube-api-access-wv4lm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:17.813163 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:17.813133 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" event={"ID":"a9eb3d46-e4da-4ec9-9a92-0b78a84796b6","Type":"ContainerStarted","Data":"985bd2af5ef27b6b23e5bf04941a82f8d5bc79dc596451bb211848654ac795b6"} Nov 03 21:15:17.816302 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:17.816280 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" Nov 03 21:15:17.816438 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:17.816328 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm" event={"ID":"4386438d-d439-49f4-8dcb-b339637c624e","Type":"ContainerDied","Data":"7b0b11c3e2b37ff5de8d37db80a1bdeec0091dcde0d6fd7063223fd542f8a246"} Nov 03 21:15:17.816438 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:17.816390 4963 scope.go:122] "RemoveContainer" containerID="6350862b1c02f5382748ca90eacdae2c756988dbe285943401aeeab7ae5858cb" Nov 03 21:15:17.983814 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:17.981442 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" Nov 03 21:15:18.107703 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:18.107614 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf1ffc34-0f1d-4022-bb27-bea026dfd5fd-kube-api-access-qzrw5\" (UniqueName: \"kubernetes.io/projected/cf1ffc34-0f1d-4022-bb27-bea026dfd5fd-kube-api-access-qzrw5\") pod \"cf1ffc34-0f1d-4022-bb27-bea026dfd5fd\" (UID: \"cf1ffc34-0f1d-4022-bb27-bea026dfd5fd\") " Nov 03 21:15:18.107885 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:18.107797 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf1ffc34-0f1d-4022-bb27-bea026dfd5fd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf1ffc34-0f1d-4022-bb27-bea026dfd5fd-maas-api-service-ca\") pod \"cf1ffc34-0f1d-4022-bb27-bea026dfd5fd\" (UID: \"cf1ffc34-0f1d-4022-bb27-bea026dfd5fd\") " Nov 03 21:15:18.108704 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:18.108658 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf1ffc34-0f1d-4022-bb27-bea026dfd5fd-maas-api-service-ca" pod "cf1ffc34-0f1d-4022-bb27-bea026dfd5fd" (UID: "cf1ffc34-0f1d-4022-bb27-bea026dfd5fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:18.110577 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:18.110548 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf1ffc34-0f1d-4022-bb27-bea026dfd5fd-kube-api-access-qzrw5" pod "cf1ffc34-0f1d-4022-bb27-bea026dfd5fd" (UID: "cf1ffc34-0f1d-4022-bb27-bea026dfd5fd"). InnerVolumeSpecName "kube-api-access-qzrw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:18.144088 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:18.144053 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm"] Nov 03 21:15:18.149625 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:18.149589 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895675-4hdpm"] Nov 03 21:15:18.210174 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:18.210143 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzrw5\" (UniqueName: \"kubernetes.io/projected/cf1ffc34-0f1d-4022-bb27-bea026dfd5fd-kube-api-access-qzrw5\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:18.210174 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:18.210170 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf1ffc34-0f1d-4022-bb27-bea026dfd5fd-maas-api-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:18.824403 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:18.824367 4963 generic.go:356] "Generic (PLEG): container finished" podID="a9eb3d46-e4da-4ec9-9a92-0b78a84796b6" containerID="985bd2af5ef27b6b23e5bf04941a82f8d5bc79dc596451bb211848654ac795b6" exitCode=60 Nov 03 21:15:18.824895 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:18.824443 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" event={"ID":"a9eb3d46-e4da-4ec9-9a92-0b78a84796b6","Type":"ContainerDied","Data":"985bd2af5ef27b6b23e5bf04941a82f8d5bc79dc596451bb211848654ac795b6"} Nov 03 21:15:18.824895 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:18.824515 4963 scope.go:122] "RemoveContainer" containerID="2e11c85e4c8c5ea290b9984884265be0a40b9f42c3f9b033507ad1f030175acb" Nov 03 21:15:18.825043 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:18.824930 4963 scope.go:122] "RemoveContainer" containerID="985bd2af5ef27b6b23e5bf04941a82f8d5bc79dc596451bb211848654ac795b6" Nov 03 21:15:18.825277 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:15:18.825255 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d_odh-ai-gateway-infra(a9eb3d46-e4da-4ec9-9a92-0b78a84796b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" podUID="a9eb3d46-e4da-4ec9-9a92-0b78a84796b6" Nov 03 21:15:18.825851 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:18.825774 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" Nov 03 21:15:18.825851 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:18.825795 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz" event={"ID":"cf1ffc34-0f1d-4022-bb27-bea026dfd5fd","Type":"ContainerDied","Data":"e2a0e779b357930d286e7ef72af8c1047d9e56de99296437f65fc382864a1b0f"} Nov 03 21:15:18.837659 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:18.837638 4963 scope.go:122] "RemoveContainer" containerID="d2f9a06adaddd909aa9952deb6f9b96a2c7dd684019d74017c46bf6cdf5cbe9b" Nov 03 21:15:18.934731 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:18.934694 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d"] Nov 03 21:15:18.974960 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:18.974926 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4386438d-d439-49f4-8dcb-b339637c624e" path="/var/lib/kubelet/pods/4386438d-d439-49f4-8dcb-b339637c624e/volumes" Nov 03 21:15:19.054537 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:19.054503 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz"] Nov 03 21:15:19.058060 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:19.058029 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895675-sn9rz"] Nov 03 21:15:20.029947 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:20.029879 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" Nov 03 21:15:20.135372 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:20.135277 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9eb3d46-e4da-4ec9-9a92-0b78a84796b6-kube-api-access-wj62j\" (UniqueName: \"kubernetes.io/projected/a9eb3d46-e4da-4ec9-9a92-0b78a84796b6-kube-api-access-wj62j\") pod \"a9eb3d46-e4da-4ec9-9a92-0b78a84796b6\" (UID: \"a9eb3d46-e4da-4ec9-9a92-0b78a84796b6\") " Nov 03 21:15:20.135615 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:20.135404 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9eb3d46-e4da-4ec9-9a92-0b78a84796b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9eb3d46-e4da-4ec9-9a92-0b78a84796b6-maas-api-service-ca\") pod \"a9eb3d46-e4da-4ec9-9a92-0b78a84796b6\" (UID: \"a9eb3d46-e4da-4ec9-9a92-0b78a84796b6\") " Nov 03 21:15:20.135857 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:20.135830 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9eb3d46-e4da-4ec9-9a92-0b78a84796b6-maas-api-service-ca" pod "a9eb3d46-e4da-4ec9-9a92-0b78a84796b6" (UID: "a9eb3d46-e4da-4ec9-9a92-0b78a84796b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:20.136129 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:20.136107 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9eb3d46-e4da-4ec9-9a92-0b78a84796b6-maas-api-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:20.138360 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:20.138324 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9eb3d46-e4da-4ec9-9a92-0b78a84796b6-kube-api-access-wj62j" pod "a9eb3d46-e4da-4ec9-9a92-0b78a84796b6" (UID: "a9eb3d46-e4da-4ec9-9a92-0b78a84796b6"). InnerVolumeSpecName "kube-api-access-wj62j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:20.238921 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:20.238872 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj62j\" (UniqueName: \"kubernetes.io/projected/a9eb3d46-e4da-4ec9-9a92-0b78a84796b6-kube-api-access-wj62j\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:20.845710 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:20.845677 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" event={"ID":"a9eb3d46-e4da-4ec9-9a92-0b78a84796b6","Type":"ContainerDied","Data":"5a14915556871b8a8bee7e726c9d01964dee3262e5610ee8dcce155ad22ee427"} Nov 03 21:15:20.845710 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:20.845699 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d" Nov 03 21:15:20.845969 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:20.845744 4963 scope.go:122] "RemoveContainer" containerID="985bd2af5ef27b6b23e5bf04941a82f8d5bc79dc596451bb211848654ac795b6" Nov 03 21:15:20.968656 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:20.968621 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf1ffc34-0f1d-4022-bb27-bea026dfd5fd" path="/var/lib/kubelet/pods/cf1ffc34-0f1d-4022-bb27-bea026dfd5fd/volumes" Nov 03 21:15:21.003765 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:21.003731 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d"] Nov 03 21:15:21.045816 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:21.045784 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895675-cc22d"] Nov 03 21:15:21.099994 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:21.099894 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:21.294804 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:21.294751 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:21.895865 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:21.895830 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:22.079940 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:22.079871 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:22.969113 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:22.969083 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9eb3d46-e4da-4ec9-9a92-0b78a84796b6" path="/var/lib/kubelet/pods/a9eb3d46-e4da-4ec9-9a92-0b78a84796b6/volumes" Nov 03 21:15:23.411454 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:23.411363 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:24.000480 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:24.000445 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:26.249241 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:26.249200 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:27.515717 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:27.515674 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:27.709768 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:27.709734 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:30.495175 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:30.495130 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:30.828499 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:30.828378 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:31.116032 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:31.115943 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:31.305362 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:31.305327 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:33.510666 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:33.510629 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:35.707276 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:35.707240 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:36.227731 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:36.227699 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw"] Nov 03 21:15:36.228422 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:36.228122 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" podUID="c368337d-2d74-4904-a9c2-4199a2aac58f" containerName="istio-proxy" containerID="cri-o://c78d2c1444429823597aa3b46dab57287b51a0b9de5fa4899930368e7cf0a427" gracePeriod=30 Nov 03 21:15:36.288916 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:36.288767 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr"] Nov 03 21:15:36.289437 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:36.289393 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" podUID="5ece95b4-8e44-47ee-9719-e2066b5f81a1" containerName="istio-proxy" containerID="cri-o://490dee731b9930b5ecb4f5a600121efc94e62de4a3b848259cbbd4f243752ad0" gracePeriod=30 Nov 03 21:15:36.650120 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:36.650031 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:37.454531 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:37.454494 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:37.640775 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:37.640738 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:38.455929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:38.452987 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:38.843031 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:38.842892 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:40.310611 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:40.310572 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:41.571304 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.571279 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:15:41.643879 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.643849 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-envoy\") pod \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " Nov 03 21:15:41.644060 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.643964 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-data\") pod \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " Nov 03 21:15:41.644060 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.644036 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-workload-socket\") pod \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " Nov 03 21:15:41.644176 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.644100 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ece95b4-8e44-47ee-9719-e2066b5f81a1-kube-api-access-brfh6\" (UniqueName: \"kubernetes.io/projected/5ece95b4-8e44-47ee-9719-e2066b5f81a1-kube-api-access-brfh6\") pod \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " Nov 03 21:15:41.644176 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.644157 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-workload-certs\") pod \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " Nov 03 21:15:41.644349 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.644192 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istiod-ca-cert\") pod \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " Nov 03 21:15:41.644349 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.644228 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-podinfo\") pod \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " Nov 03 21:15:41.644349 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.644282 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-data" pod "5ece95b4-8e44-47ee-9719-e2066b5f81a1" (UID: "5ece95b4-8e44-47ee-9719-e2066b5f81a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:41.644349 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.644327 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-token\" (UniqueName: \"kubernetes.io/projected/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-token\") pod \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " Nov 03 21:15:41.644649 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.644368 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-credential-socket\") pod \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\" (UID: \"5ece95b4-8e44-47ee-9719-e2066b5f81a1\") " Nov 03 21:15:41.644649 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.644433 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-workload-socket" pod "5ece95b4-8e44-47ee-9719-e2066b5f81a1" (UID: "5ece95b4-8e44-47ee-9719-e2066b5f81a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:41.644649 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.644492 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-workload-certs" pod "5ece95b4-8e44-47ee-9719-e2066b5f81a1" (UID: "5ece95b4-8e44-47ee-9719-e2066b5f81a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:41.644649 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.644600 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istiod-ca-cert" pod "5ece95b4-8e44-47ee-9719-e2066b5f81a1" (UID: "5ece95b4-8e44-47ee-9719-e2066b5f81a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:41.644989 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.644812 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-credential-socket" pod "5ece95b4-8e44-47ee-9719-e2066b5f81a1" (UID: "5ece95b4-8e44-47ee-9719-e2066b5f81a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:41.645076 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.645054 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.645158 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.645076 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.645158 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.645091 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.645158 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.645106 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.645158 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.645120 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.646796 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.646769 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-envoy" pod "5ece95b4-8e44-47ee-9719-e2066b5f81a1" (UID: "5ece95b4-8e44-47ee-9719-e2066b5f81a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:41.646892 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.646807 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-podinfo" pod "5ece95b4-8e44-47ee-9719-e2066b5f81a1" (UID: "5ece95b4-8e44-47ee-9719-e2066b5f81a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:15:41.646986 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.646961 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ece95b4-8e44-47ee-9719-e2066b5f81a1-kube-api-access-brfh6" pod "5ece95b4-8e44-47ee-9719-e2066b5f81a1" (UID: "5ece95b4-8e44-47ee-9719-e2066b5f81a1"). InnerVolumeSpecName "kube-api-access-brfh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:41.647079 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.647054 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-token" pod "5ece95b4-8e44-47ee-9719-e2066b5f81a1" (UID: "5ece95b4-8e44-47ee-9719-e2066b5f81a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:41.749946 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.747298 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brfh6\" (UniqueName: \"kubernetes.io/projected/5ece95b4-8e44-47ee-9719-e2066b5f81a1-kube-api-access-brfh6\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.749946 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.747338 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.749946 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.747354 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.749946 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.747368 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ece95b4-8e44-47ee-9719-e2066b5f81a1-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.917386 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:41.917360 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:15:42.008336 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.008293 4963 generic.go:356] "Generic (PLEG): container finished" podID="c368337d-2d74-4904-a9c2-4199a2aac58f" containerID="c78d2c1444429823597aa3b46dab57287b51a0b9de5fa4899930368e7cf0a427" exitCode=0 Nov 03 21:15:42.008549 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.008475 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" event={"ID":"c368337d-2d74-4904-a9c2-4199a2aac58f","Type":"ContainerDied","Data":"c78d2c1444429823597aa3b46dab57287b51a0b9de5fa4899930368e7cf0a427"} Nov 03 21:15:42.008615 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.008567 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" event={"ID":"c368337d-2d74-4904-a9c2-4199a2aac58f","Type":"ContainerDied","Data":"75671f78972c5f5da25856669bfc6d6fe5ef5a470b994ec87725405360af9b36"} Nov 03 21:15:42.008669 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.008604 4963 scope.go:122] "RemoveContainer" containerID="c78d2c1444429823597aa3b46dab57287b51a0b9de5fa4899930368e7cf0a427" Nov 03 21:15:42.009241 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.008815 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" Nov 03 21:15:42.011303 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.011266 4963 generic.go:356] "Generic (PLEG): container finished" podID="5ece95b4-8e44-47ee-9719-e2066b5f81a1" containerID="490dee731b9930b5ecb4f5a600121efc94e62de4a3b848259cbbd4f243752ad0" exitCode=0 Nov 03 21:15:42.011528 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.011510 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" event={"ID":"5ece95b4-8e44-47ee-9719-e2066b5f81a1","Type":"ContainerDied","Data":"490dee731b9930b5ecb4f5a600121efc94e62de4a3b848259cbbd4f243752ad0"} Nov 03 21:15:42.011581 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.011556 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" event={"ID":"5ece95b4-8e44-47ee-9719-e2066b5f81a1","Type":"ContainerDied","Data":"971ce7749e57a9d25f28359827b25988a25a93990fca42b5c4ab61d8a25995d2"} Nov 03 21:15:42.011581 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.011565 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr" Nov 03 21:15:42.024579 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.024558 4963 scope.go:122] "RemoveContainer" containerID="c78d2c1444429823597aa3b46dab57287b51a0b9de5fa4899930368e7cf0a427" Nov 03 21:15:42.025025 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:15:42.024984 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c78d2c1444429823597aa3b46dab57287b51a0b9de5fa4899930368e7cf0a427\": container with ID starting with c78d2c1444429823597aa3b46dab57287b51a0b9de5fa4899930368e7cf0a427 not found: ID does not exist" containerID="c78d2c1444429823597aa3b46dab57287b51a0b9de5fa4899930368e7cf0a427" Nov 03 21:15:42.025091 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.025052 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c78d2c1444429823597aa3b46dab57287b51a0b9de5fa4899930368e7cf0a427"} err="failed to get container status \"c78d2c1444429823597aa3b46dab57287b51a0b9de5fa4899930368e7cf0a427\": rpc error: code = NotFound desc = could not find container \"c78d2c1444429823597aa3b46dab57287b51a0b9de5fa4899930368e7cf0a427\": container with ID starting with c78d2c1444429823597aa3b46dab57287b51a0b9de5fa4899930368e7cf0a427 not found: ID does not exist" Nov 03 21:15:42.025153 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.025089 4963 scope.go:122] "RemoveContainer" containerID="490dee731b9930b5ecb4f5a600121efc94e62de4a3b848259cbbd4f243752ad0" Nov 03 21:15:42.035357 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.035329 4963 scope.go:122] "RemoveContainer" containerID="490dee731b9930b5ecb4f5a600121efc94e62de4a3b848259cbbd4f243752ad0" Nov 03 21:15:42.036006 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:15:42.035974 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490dee731b9930b5ecb4f5a600121efc94e62de4a3b848259cbbd4f243752ad0\": container with ID starting with 490dee731b9930b5ecb4f5a600121efc94e62de4a3b848259cbbd4f243752ad0 not found: ID does not exist" containerID="490dee731b9930b5ecb4f5a600121efc94e62de4a3b848259cbbd4f243752ad0" Nov 03 21:15:42.036163 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.036016 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490dee731b9930b5ecb4f5a600121efc94e62de4a3b848259cbbd4f243752ad0"} err="failed to get container status \"490dee731b9930b5ecb4f5a600121efc94e62de4a3b848259cbbd4f243752ad0\": rpc error: code = NotFound desc = could not find container \"490dee731b9930b5ecb4f5a600121efc94e62de4a3b848259cbbd4f243752ad0\": container with ID starting with 490dee731b9930b5ecb4f5a600121efc94e62de4a3b848259cbbd4f243752ad0 not found: ID does not exist" Nov 03 21:15:42.051016 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.050982 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c368337d-2d74-4904-a9c2-4199a2aac58f-kube-api-access-s4ncv\" (UniqueName: \"kubernetes.io/projected/c368337d-2d74-4904-a9c2-4199a2aac58f-kube-api-access-s4ncv\") pod \"c368337d-2d74-4904-a9c2-4199a2aac58f\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " Nov 03 21:15:42.051208 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.051033 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-envoy\") pod \"c368337d-2d74-4904-a9c2-4199a2aac58f\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " Nov 03 21:15:42.051208 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.051058 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-data\") pod \"c368337d-2d74-4904-a9c2-4199a2aac58f\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " Nov 03 21:15:42.051208 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.051079 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-token\" (UniqueName: \"kubernetes.io/projected/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-token\") pod \"c368337d-2d74-4904-a9c2-4199a2aac58f\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " Nov 03 21:15:42.051208 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.051114 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c368337d-2d74-4904-a9c2-4199a2aac58f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c368337d-2d74-4904-a9c2-4199a2aac58f-istiod-ca-cert\") pod \"c368337d-2d74-4904-a9c2-4199a2aac58f\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " Nov 03 21:15:42.051208 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.051181 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-workload-certs\") pod \"c368337d-2d74-4904-a9c2-4199a2aac58f\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " Nov 03 21:15:42.051476 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.051227 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-podinfo\") pod \"c368337d-2d74-4904-a9c2-4199a2aac58f\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " Nov 03 21:15:42.051476 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.051273 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-credential-socket\") pod \"c368337d-2d74-4904-a9c2-4199a2aac58f\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " Nov 03 21:15:42.051476 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.051441 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-workload-socket\") pod \"c368337d-2d74-4904-a9c2-4199a2aac58f\" (UID: \"c368337d-2d74-4904-a9c2-4199a2aac58f\") " Nov 03 21:15:42.051709 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.051675 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-workload-certs" pod "c368337d-2d74-4904-a9c2-4199a2aac58f" (UID: "c368337d-2d74-4904-a9c2-4199a2aac58f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:42.051709 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.051701 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-data" pod "c368337d-2d74-4904-a9c2-4199a2aac58f" (UID: "c368337d-2d74-4904-a9c2-4199a2aac58f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:42.052098 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.051951 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-workload-socket" pod "c368337d-2d74-4904-a9c2-4199a2aac58f" (UID: "c368337d-2d74-4904-a9c2-4199a2aac58f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:42.052168 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.052078 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:42.052168 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.052127 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:42.052168 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.052143 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:42.052168 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.052150 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-credential-socket" pod "c368337d-2d74-4904-a9c2-4199a2aac58f" (UID: "c368337d-2d74-4904-a9c2-4199a2aac58f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:42.052576 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.052554 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c368337d-2d74-4904-a9c2-4199a2aac58f-istiod-ca-cert" pod "c368337d-2d74-4904-a9c2-4199a2aac58f" (UID: "c368337d-2d74-4904-a9c2-4199a2aac58f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:42.054114 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.054082 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c368337d-2d74-4904-a9c2-4199a2aac58f-kube-api-access-s4ncv" pod "c368337d-2d74-4904-a9c2-4199a2aac58f" (UID: "c368337d-2d74-4904-a9c2-4199a2aac58f"). InnerVolumeSpecName "kube-api-access-s4ncv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:42.054348 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.054331 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-token" pod "c368337d-2d74-4904-a9c2-4199a2aac58f" (UID: "c368337d-2d74-4904-a9c2-4199a2aac58f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:42.054551 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.054526 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-envoy" pod "c368337d-2d74-4904-a9c2-4199a2aac58f" (UID: "c368337d-2d74-4904-a9c2-4199a2aac58f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:42.054717 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.054693 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-podinfo" pod "c368337d-2d74-4904-a9c2-4199a2aac58f" (UID: "c368337d-2d74-4904-a9c2-4199a2aac58f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:15:42.154241 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.154145 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4ncv\" (UniqueName: \"kubernetes.io/projected/c368337d-2d74-4904-a9c2-4199a2aac58f-kube-api-access-s4ncv\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:42.154241 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.154181 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:42.154241 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.154195 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:42.154241 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.154207 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c368337d-2d74-4904-a9c2-4199a2aac58f-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:42.154241 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.154221 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c368337d-2d74-4904-a9c2-4199a2aac58f-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:42.154241 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.154234 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c368337d-2d74-4904-a9c2-4199a2aac58f-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:42.191915 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.191862 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr"] Nov 03 21:15:42.195456 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.195430 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-xfshr"] Nov 03 21:15:42.427890 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.427788 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw"] Nov 03 21:15:42.436652 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.436616 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw"] Nov 03 21:15:42.527985 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:15:42.527949 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:15:42.597231 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.597201 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:42.785333 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.785294 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:42.869155 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.869114 4963 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:15:42.869361 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.869203 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-5gzzw" podUID="c368337d-2d74-4904-a9c2-4199a2aac58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:15:42.968883 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.968848 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ece95b4-8e44-47ee-9719-e2066b5f81a1" path="/var/lib/kubelet/pods/5ece95b4-8e44-47ee-9719-e2066b5f81a1/volumes" Nov 03 21:15:42.969354 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:42.969339 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c368337d-2d74-4904-a9c2-4199a2aac58f" path="/var/lib/kubelet/pods/c368337d-2d74-4904-a9c2-4199a2aac58f/volumes" Nov 03 21:15:44.323580 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:44.323531 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv"] Nov 03 21:15:44.324086 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:44.323803 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" podUID="107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e" containerName="maas-api" containerID="cri-o://4e8890db5848c1bf81557302004c6b03c05b605b66905851457022895fbf1e85" gracePeriod=30 Nov 03 21:15:44.463598 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:44.463559 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:44.647416 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:44.647367 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:44.657845 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:44.657807 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:15:44.788505 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:44.788434 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-kube-api-access-bm2bp\" (UniqueName: \"kubernetes.io/projected/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-kube-api-access-bm2bp\") pod \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\" (UID: \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\") " Nov 03 21:15:44.788505 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:44.788484 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-maas-api-tls\") pod \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\" (UID: \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\") " Nov 03 21:15:44.788768 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:44.788535 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-openshift-service-ca\") pod \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\" (UID: \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\") " Nov 03 21:15:44.788768 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:44.788636 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-metrics-certs\") pod \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\" (UID: \"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e\") " Nov 03 21:15:44.790760 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:44.790148 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-openshift-service-ca" pod "107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e" (UID: "107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:44.803002 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:44.802964 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-kube-api-access-bm2bp" pod "107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e" (UID: "107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e"). InnerVolumeSpecName "kube-api-access-bm2bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:44.806174 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:44.806139 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-metrics-certs" pod "107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e" (UID: "107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:44.806174 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:44.806165 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-maas-api-tls" pod "107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e" (UID: "107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:44.892474 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:44.892385 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm2bp\" (UniqueName: \"kubernetes.io/projected/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-kube-api-access-bm2bp\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:44.892474 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:44.892417 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:44.892474 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:44.892428 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:44.892474 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:44.892439 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:45.036605 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:45.036567 4963 generic.go:356] "Generic (PLEG): container finished" podID="107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e" containerID="4e8890db5848c1bf81557302004c6b03c05b605b66905851457022895fbf1e85" exitCode=0 Nov 03 21:15:45.036757 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:45.036616 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" Nov 03 21:15:45.036757 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:45.036659 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" event={"ID":"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e","Type":"ContainerDied","Data":"4e8890db5848c1bf81557302004c6b03c05b605b66905851457022895fbf1e85"} Nov 03 21:15:45.036757 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:45.036721 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv" event={"ID":"107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e","Type":"ContainerDied","Data":"3575eaf0fe4b0b39d7105a2d854ea99ea470a879d656ef65c854413edb33d1b9"} Nov 03 21:15:45.036757 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:45.036746 4963 scope.go:122] "RemoveContainer" containerID="4e8890db5848c1bf81557302004c6b03c05b605b66905851457022895fbf1e85" Nov 03 21:15:45.046481 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:45.046462 4963 scope.go:122] "RemoveContainer" containerID="4e8890db5848c1bf81557302004c6b03c05b605b66905851457022895fbf1e85" Nov 03 21:15:45.046788 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:15:45.046765 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e8890db5848c1bf81557302004c6b03c05b605b66905851457022895fbf1e85\": container with ID starting with 4e8890db5848c1bf81557302004c6b03c05b605b66905851457022895fbf1e85 not found: ID does not exist" containerID="4e8890db5848c1bf81557302004c6b03c05b605b66905851457022895fbf1e85" Nov 03 21:15:45.046840 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:45.046805 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e8890db5848c1bf81557302004c6b03c05b605b66905851457022895fbf1e85"} err="failed to get container status \"4e8890db5848c1bf81557302004c6b03c05b605b66905851457022895fbf1e85\": rpc error: code = NotFound desc = could not find container \"4e8890db5848c1bf81557302004c6b03c05b605b66905851457022895fbf1e85\": container with ID starting with 4e8890db5848c1bf81557302004c6b03c05b605b66905851457022895fbf1e85 not found: ID does not exist" Nov 03 21:15:45.196017 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:45.195984 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv"] Nov 03 21:15:45.243194 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:45.243159 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e92e3-77778b558d-6xbtv"] Nov 03 21:15:45.822577 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:45.822537 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:46.012335 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:46.012294 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:46.300527 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:46.300491 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:46.510925 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:46.510473 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:46.968216 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:46.968183 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e" path="/var/lib/kubelet/pods/107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e/volumes" Nov 03 21:15:48.734655 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:48.734626 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:48.934808 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:48.934778 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:50.220462 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:50.220423 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:50.729364 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:50.729329 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:50.925930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:50.924273 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:51.118705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:51.118618 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:51.520076 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:51.520040 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:55.909634 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:55.909600 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:57.817815 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:57.817780 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:58.929625 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:58.929580 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:59.114520 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:15:59.114487 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:00.916142 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:00.916112 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:03.665241 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:03.665203 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:05.650427 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:05.650392 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:10.207464 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:10.207424 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:10.417762 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:10.417718 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:10.611566 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:10.611477 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:12.528653 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:16:12.528618 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:16:15.161387 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:15.161354 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:17.460476 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.460440 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7481-4ff374"] Nov 03 21:16:17.461163 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461138 4963 cpu_manager.go:414] "containerMap: removing container" podUID="a9eb3d46-e4da-4ec9-9a92-0b78a84796b6" containerName="cleanup" Nov 03 21:16:17.461163 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461158 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9eb3d46-e4da-4ec9-9a92-0b78a84796b6" containerName="cleanup" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461173 4963 cpu_manager.go:414] "containerMap: removing container" podUID="4386438d-d439-49f4-8dcb-b339637c624e" containerName="cleanup" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461179 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4386438d-d439-49f4-8dcb-b339637c624e" containerName="cleanup" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461191 4963 cpu_manager.go:414] "containerMap: removing container" podUID="107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e" containerName="maas-api" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461196 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e" containerName="maas-api" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461206 4963 cpu_manager.go:414] "containerMap: removing container" podUID="a9eb3d46-e4da-4ec9-9a92-0b78a84796b6" containerName="cleanup" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461211 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9eb3d46-e4da-4ec9-9a92-0b78a84796b6" containerName="cleanup" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461219 4963 cpu_manager.go:414] "containerMap: removing container" podUID="cf1ffc34-0f1d-4022-bb27-bea026dfd5fd" containerName="cleanup" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461223 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf1ffc34-0f1d-4022-bb27-bea026dfd5fd" containerName="cleanup" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461230 4963 cpu_manager.go:414] "containerMap: removing container" podUID="cf1ffc34-0f1d-4022-bb27-bea026dfd5fd" containerName="cleanup" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461234 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf1ffc34-0f1d-4022-bb27-bea026dfd5fd" containerName="cleanup" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461247 4963 cpu_manager.go:414] "containerMap: removing container" podUID="5ece95b4-8e44-47ee-9719-e2066b5f81a1" containerName="istio-proxy" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461251 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ece95b4-8e44-47ee-9719-e2066b5f81a1" containerName="istio-proxy" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461258 4963 cpu_manager.go:414] "containerMap: removing container" podUID="a9eb3d46-e4da-4ec9-9a92-0b78a84796b6" containerName="cleanup" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461262 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9eb3d46-e4da-4ec9-9a92-0b78a84796b6" containerName="cleanup" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461270 4963 cpu_manager.go:414] "containerMap: removing container" podUID="4386438d-d439-49f4-8dcb-b339637c624e" containerName="cleanup" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461274 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4386438d-d439-49f4-8dcb-b339637c624e" containerName="cleanup" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461280 4963 cpu_manager.go:414] "containerMap: removing container" podUID="cf1ffc34-0f1d-4022-bb27-bea026dfd5fd" containerName="cleanup" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461285 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf1ffc34-0f1d-4022-bb27-bea026dfd5fd" containerName="cleanup" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461301 4963 cpu_manager.go:414] "containerMap: removing container" podUID="4386438d-d439-49f4-8dcb-b339637c624e" containerName="cleanup" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461307 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4386438d-d439-49f4-8dcb-b339637c624e" containerName="cleanup" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461317 4963 cpu_manager.go:414] "containerMap: removing container" podUID="c368337d-2d74-4904-a9c2-4199a2aac58f" containerName="istio-proxy" Nov 03 21:16:17.461331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461321 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c368337d-2d74-4904-a9c2-4199a2aac58f" containerName="istio-proxy" Nov 03 21:16:17.462018 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461420 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9eb3d46-e4da-4ec9-9a92-0b78a84796b6" containerName="cleanup" Nov 03 21:16:17.462018 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461429 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="4386438d-d439-49f4-8dcb-b339637c624e" containerName="cleanup" Nov 03 21:16:17.462018 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461436 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="107dcc3b-5c84-4d4c-a5a4-697fe8b4e07e" containerName="maas-api" Nov 03 21:16:17.462018 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461441 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9eb3d46-e4da-4ec9-9a92-0b78a84796b6" containerName="cleanup" Nov 03 21:16:17.462018 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461449 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf1ffc34-0f1d-4022-bb27-bea026dfd5fd" containerName="cleanup" Nov 03 21:16:17.462018 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461454 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf1ffc34-0f1d-4022-bb27-bea026dfd5fd" containerName="cleanup" Nov 03 21:16:17.462018 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461461 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ece95b4-8e44-47ee-9719-e2066b5f81a1" containerName="istio-proxy" Nov 03 21:16:17.462018 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461473 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="4386438d-d439-49f4-8dcb-b339637c624e" containerName="cleanup" Nov 03 21:16:17.462018 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461480 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="4386438d-d439-49f4-8dcb-b339637c624e" containerName="cleanup" Nov 03 21:16:17.462018 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.461486 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="c368337d-2d74-4904-a9c2-4199a2aac58f" containerName="istio-proxy" Nov 03 21:16:17.465104 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.465087 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-4ff374" Nov 03 21:16:17.514894 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.514847 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7vxf5\"" Nov 03 21:16:17.517018 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.516989 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-4ff374"] Nov 03 21:16:17.537501 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.537458 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg5d2\" (UniqueName: \"kubernetes.io/projected/8088dc64-b0ae-4311-bf77-1c621e54cabb-kube-api-access-xg5d2\") pod \"test-curl-7481-4ff374\" (UID: \"8088dc64-b0ae-4311-bf77-1c621e54cabb\") " pod="opendatahub/test-curl-7481-4ff374" Nov 03 21:16:17.638385 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.638349 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg5d2\" (UniqueName: \"kubernetes.io/projected/8088dc64-b0ae-4311-bf77-1c621e54cabb-kube-api-access-xg5d2\") pod \"test-curl-7481-4ff374\" (UID: \"8088dc64-b0ae-4311-bf77-1c621e54cabb\") " pod="opendatahub/test-curl-7481-4ff374" Nov 03 21:16:17.698188 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.698153 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg5d2\" (UniqueName: \"kubernetes.io/projected/8088dc64-b0ae-4311-bf77-1c621e54cabb-kube-api-access-xg5d2\") pod \"test-curl-7481-4ff374\" (UID: \"8088dc64-b0ae-4311-bf77-1c621e54cabb\") " pod="opendatahub/test-curl-7481-4ff374" Nov 03 21:16:17.777943 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:17.777820 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-4ff374" Nov 03 21:16:18.154610 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:18.154528 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-4ff374"] Nov 03 21:16:18.159932 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:16:18.159880 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8088dc64_b0ae_4311_bf77_1c621e54cabb.slice/crio-908bd6666f0daea534727169978090573995b3798656fc60592bdffbf35fd2ed WatchSource:0}: Error finding container 908bd6666f0daea534727169978090573995b3798656fc60592bdffbf35fd2ed: Status 404 returned error can't find the container with id 908bd6666f0daea534727169978090573995b3798656fc60592bdffbf35fd2ed Nov 03 21:16:18.264256 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:18.264217 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-4ff374" event={"ID":"8088dc64-b0ae-4311-bf77-1c621e54cabb","Type":"ContainerStarted","Data":"834d8e154756fd25a6f04c5721e46d208b4cd7a32959de3aa4e6e72b8d263c20"} Nov 03 21:16:18.264256 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:18.264262 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-4ff374" event={"ID":"8088dc64-b0ae-4311-bf77-1c621e54cabb","Type":"ContainerStarted","Data":"908bd6666f0daea534727169978090573995b3798656fc60592bdffbf35fd2ed"} Nov 03 21:16:18.373630 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:18.373540 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7481-4ff374" podStartSLOduration=1.373514988 podStartE2EDuration="1.373514988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:16:18.372108855 +0000 UTC m=+1727.532181128" watchObservedRunningTime="2026-11-03 21:16:18.373514988 +0000 UTC m=+1727.533587265" Nov 03 21:16:19.880055 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:19.880018 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7481-4ff374"] Nov 03 21:16:19.884634 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:19.884468 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7481-4ff374"] Nov 03 21:16:20.049036 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:20.048993 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:20.277362 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:20.277326 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7481-4ff374" podUID="8088dc64-b0ae-4311-bf77-1c621e54cabb" containerName="test-curl-7481-4ff374" containerID="cri-o://834d8e154756fd25a6f04c5721e46d208b4cd7a32959de3aa4e6e72b8d263c20" gracePeriod=2 Nov 03 21:16:20.326101 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:16:20.326055 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7481-4ff374\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="8088dc64-b0ae-4311-bf77-1c621e54cabb" pod="opendatahub/test-curl-7481-4ff374" Nov 03 21:16:20.457972 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:20.457937 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:21.012105 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:16:21.012066 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7481-4ff374\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="8088dc64-b0ae-4311-bf77-1c621e54cabb" pod="opendatahub/test-curl-7481-4ff374" Nov 03 21:16:22.522591 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:22.522558 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-4ff374" Nov 03 21:16:22.602051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:22.601963 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8088dc64-b0ae-4311-bf77-1c621e54cabb-kube-api-access-xg5d2\" (UniqueName: \"kubernetes.io/projected/8088dc64-b0ae-4311-bf77-1c621e54cabb-kube-api-access-xg5d2\") pod \"8088dc64-b0ae-4311-bf77-1c621e54cabb\" (UID: \"8088dc64-b0ae-4311-bf77-1c621e54cabb\") " Nov 03 21:16:22.604144 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:22.604117 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8088dc64-b0ae-4311-bf77-1c621e54cabb-kube-api-access-xg5d2" pod "8088dc64-b0ae-4311-bf77-1c621e54cabb" (UID: "8088dc64-b0ae-4311-bf77-1c621e54cabb"). InnerVolumeSpecName "kube-api-access-xg5d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:22.703321 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:22.703283 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg5d2\" (UniqueName: \"kubernetes.io/projected/8088dc64-b0ae-4311-bf77-1c621e54cabb-kube-api-access-xg5d2\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:22.967922 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:22.967876 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8088dc64-b0ae-4311-bf77-1c621e54cabb" path="/var/lib/kubelet/pods/8088dc64-b0ae-4311-bf77-1c621e54cabb/volumes" Nov 03 21:16:23.302039 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:23.301848 4963 generic.go:356] "Generic (PLEG): container finished" podID="8088dc64-b0ae-4311-bf77-1c621e54cabb" containerID="834d8e154756fd25a6f04c5721e46d208b4cd7a32959de3aa4e6e72b8d263c20" exitCode=137 Nov 03 21:16:23.302039 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:23.301984 4963 scope.go:122] "RemoveContainer" containerID="834d8e154756fd25a6f04c5721e46d208b4cd7a32959de3aa4e6e72b8d263c20" Nov 03 21:16:23.302280 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:23.302062 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-4ff374" Nov 03 21:16:23.315140 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:23.315108 4963 scope.go:122] "RemoveContainer" containerID="834d8e154756fd25a6f04c5721e46d208b4cd7a32959de3aa4e6e72b8d263c20" Nov 03 21:16:23.315703 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:16:23.315675 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834d8e154756fd25a6f04c5721e46d208b4cd7a32959de3aa4e6e72b8d263c20\": container with ID starting with 834d8e154756fd25a6f04c5721e46d208b4cd7a32959de3aa4e6e72b8d263c20 not found: ID does not exist" containerID="834d8e154756fd25a6f04c5721e46d208b4cd7a32959de3aa4e6e72b8d263c20" Nov 03 21:16:23.315814 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:23.315711 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834d8e154756fd25a6f04c5721e46d208b4cd7a32959de3aa4e6e72b8d263c20"} err="failed to get container status \"834d8e154756fd25a6f04c5721e46d208b4cd7a32959de3aa4e6e72b8d263c20\": rpc error: code = NotFound desc = could not find container \"834d8e154756fd25a6f04c5721e46d208b4cd7a32959de3aa4e6e72b8d263c20\": container with ID starting with 834d8e154756fd25a6f04c5721e46d208b4cd7a32959de3aa4e6e72b8d263c20 not found: ID does not exist" Nov 03 21:16:23.395807 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:23.395775 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:23.854270 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:23.854221 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:24.460110 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:24.460074 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt"] Nov 03 21:16:24.460727 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:24.460711 4963 cpu_manager.go:414] "containerMap: removing container" podUID="8088dc64-b0ae-4311-bf77-1c621e54cabb" containerName="test-curl-7481-4ff374" Nov 03 21:16:24.460778 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:24.460728 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8088dc64-b0ae-4311-bf77-1c621e54cabb" containerName="test-curl-7481-4ff374" Nov 03 21:16:24.460832 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:24.460821 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9eb3d46-e4da-4ec9-9a92-0b78a84796b6" containerName="cleanup" Nov 03 21:16:24.460873 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:24.460837 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf1ffc34-0f1d-4022-bb27-bea026dfd5fd" containerName="cleanup" Nov 03 21:16:24.460873 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:24.460855 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="8088dc64-b0ae-4311-bf77-1c621e54cabb" containerName="test-curl-7481-4ff374" Nov 03 21:16:24.465928 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:24.465885 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt" Nov 03 21:16:24.519307 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:24.519274 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt"] Nov 03 21:16:24.528513 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:24.528478 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67c6469-1e35-423a-8428-e19f3f7bceb7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt\" (UID: \"d67c6469-1e35-423a-8428-e19f3f7bceb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt" Nov 03 21:16:24.528697 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:24.528607 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mhfd\" (UniqueName: \"kubernetes.io/projected/d67c6469-1e35-423a-8428-e19f3f7bceb7-kube-api-access-4mhfd\") pod \"payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt\" (UID: \"d67c6469-1e35-423a-8428-e19f3f7bceb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt" Nov 03 21:16:24.562602 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:24.562568 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0f8726\"" Nov 03 21:16:24.562602 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:24.562581 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0f8726-dockercfg-zkhlj\"" Nov 03 21:16:24.630421 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:24.630387 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhfd\" (UniqueName: \"kubernetes.io/projected/d67c6469-1e35-423a-8428-e19f3f7bceb7-kube-api-access-4mhfd\") pod \"payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt\" (UID: \"d67c6469-1e35-423a-8428-e19f3f7bceb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt" Nov 03 21:16:24.630638 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:24.630518 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67c6469-1e35-423a-8428-e19f3f7bceb7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt\" (UID: \"d67c6469-1e35-423a-8428-e19f3f7bceb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt" Nov 03 21:16:24.631259 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:24.631240 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67c6469-1e35-423a-8428-e19f3f7bceb7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt\" (UID: \"d67c6469-1e35-423a-8428-e19f3f7bceb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt" Nov 03 21:16:24.689459 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:24.689427 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mhfd\" (UniqueName: \"kubernetes.io/projected/d67c6469-1e35-423a-8428-e19f3f7bceb7-kube-api-access-4mhfd\") pod \"payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt\" (UID: \"d67c6469-1e35-423a-8428-e19f3f7bceb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt" Nov 03 21:16:24.780655 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:24.780563 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt" Nov 03 21:16:25.154507 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:25.154408 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt"] Nov 03 21:16:25.159400 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:16:25.159366 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67c6469_1e35_423a_8428_e19f3f7bceb7.slice/crio-2df33f5ec347c491ab99bdfade524cc8f3d4e6f594593e02ba76456663ebc58c WatchSource:0}: Error finding container 2df33f5ec347c491ab99bdfade524cc8f3d4e6f594593e02ba76456663ebc58c: Status 404 returned error can't find the container with id 2df33f5ec347c491ab99bdfade524cc8f3d4e6f594593e02ba76456663ebc58c Nov 03 21:16:25.161504 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:25.161486 4963 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:16:25.318887 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:25.318849 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt" event={"ID":"d67c6469-1e35-423a-8428-e19f3f7bceb7","Type":"ContainerStarted","Data":"2df33f5ec347c491ab99bdfade524cc8f3d4e6f594593e02ba76456663ebc58c"} Nov 03 21:16:25.377981 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:25.377936 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx"] Nov 03 21:16:25.492816 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:25.492767 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:26.259677 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:26.259634 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:28.382208 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:28.382169 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt" event={"ID":"d67c6469-1e35-423a-8428-e19f3f7bceb7","Type":"ContainerStarted","Data":"c968a3f9b81f74f2debf8ef31e306895e591d08fac3f3c10c246d9b6aa6d84f0"} Nov 03 21:16:28.382597 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:28.382308 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt" Nov 03 21:16:28.496089 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:28.496033 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt" podStartSLOduration=1.8413782090000002 podStartE2EDuration="4.496019095s" totalImagesPullingTime="2.654640886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:16:28.494845123 +0000 UTC m=+1737.654917396" watchObservedRunningTime="2026-11-03 21:16:28.496019095 +0000 UTC m=+1737.656091366" Nov 03 21:16:30.818613 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:30.818577 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:30.997387 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:30.997353 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:35.321148 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:35.321109 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:39.393919 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:39.393878 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt" Nov 03 21:16:39.652814 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:39.652718 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:39.853717 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:39.853685 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:40.042202 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.042127 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:40.399288 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.399204 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" podUID="f315250f-0dc8-4b98-bb9a-9d998e62ac01" containerName="main" containerID="cri-o://882fbddf3e2e325f6bc9a26268f0999404e8a02f73a5b77259e6ef0bff10c539" gracePeriod=45 Nov 03 21:16:40.668381 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.668314 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:16:40.752546 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.752510 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-model-cache\") pod \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " Nov 03 21:16:40.752725 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.752652 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f315250f-0dc8-4b98-bb9a-9d998e62ac01-kube-api-access-lq76r\" (UniqueName: \"kubernetes.io/projected/f315250f-0dc8-4b98-bb9a-9d998e62ac01-kube-api-access-lq76r\") pod \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " Nov 03 21:16:40.752725 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.752690 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-home\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-home\") pod \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " Nov 03 21:16:40.752804 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.752749 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f315250f-0dc8-4b98-bb9a-9d998e62ac01-tls-certs\" (UniqueName: \"kubernetes.io/secret/f315250f-0dc8-4b98-bb9a-9d998e62ac01-tls-certs\") pod \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " Nov 03 21:16:40.752847 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.752828 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-dshm\") pod \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " Nov 03 21:16:40.752847 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.752830 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-model-cache" pod "f315250f-0dc8-4b98-bb9a-9d998e62ac01" (UID: "f315250f-0dc8-4b98-bb9a-9d998e62ac01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:40.752958 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.752870 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-tmp-dir\") pod \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\" (UID: \"f315250f-0dc8-4b98-bb9a-9d998e62ac01\") " Nov 03 21:16:40.753062 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.753037 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-home" pod "f315250f-0dc8-4b98-bb9a-9d998e62ac01" (UID: "f315250f-0dc8-4b98-bb9a-9d998e62ac01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:40.753193 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.753169 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-tmp-dir" pod "f315250f-0dc8-4b98-bb9a-9d998e62ac01" (UID: "f315250f-0dc8-4b98-bb9a-9d998e62ac01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:40.753445 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.753426 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:40.753526 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.753452 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:40.753526 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.753482 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:40.755337 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.755188 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-dshm" pod "f315250f-0dc8-4b98-bb9a-9d998e62ac01" (UID: "f315250f-0dc8-4b98-bb9a-9d998e62ac01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:40.755337 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.755251 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f315250f-0dc8-4b98-bb9a-9d998e62ac01-kube-api-access-lq76r" pod "f315250f-0dc8-4b98-bb9a-9d998e62ac01" (UID: "f315250f-0dc8-4b98-bb9a-9d998e62ac01"). InnerVolumeSpecName "kube-api-access-lq76r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:40.755337 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.755303 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f315250f-0dc8-4b98-bb9a-9d998e62ac01-tls-certs" pod "f315250f-0dc8-4b98-bb9a-9d998e62ac01" (UID: "f315250f-0dc8-4b98-bb9a-9d998e62ac01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:40.854758 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.854724 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq76r\" (UniqueName: \"kubernetes.io/projected/f315250f-0dc8-4b98-bb9a-9d998e62ac01-kube-api-access-lq76r\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:40.854758 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.854754 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f315250f-0dc8-4b98-bb9a-9d998e62ac01-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:40.854758 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:40.854764 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f315250f-0dc8-4b98-bb9a-9d998e62ac01-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:41.478848 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:41.478800 4963 generic.go:356] "Generic (PLEG): container finished" podID="f315250f-0dc8-4b98-bb9a-9d998e62ac01" containerID="882fbddf3e2e325f6bc9a26268f0999404e8a02f73a5b77259e6ef0bff10c539" exitCode=0 Nov 03 21:16:41.479355 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:41.478857 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" event={"ID":"f315250f-0dc8-4b98-bb9a-9d998e62ac01","Type":"ContainerDied","Data":"882fbddf3e2e325f6bc9a26268f0999404e8a02f73a5b77259e6ef0bff10c539"} Nov 03 21:16:41.479355 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:41.478868 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" Nov 03 21:16:41.479355 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:41.478897 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx" event={"ID":"f315250f-0dc8-4b98-bb9a-9d998e62ac01","Type":"ContainerDied","Data":"0025c8a6f3fff9b71c2f62afa3e364775d5f3c3bc04e7f3a998741a9dc714010"} Nov 03 21:16:41.479355 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:41.478936 4963 scope.go:122] "RemoveContainer" containerID="882fbddf3e2e325f6bc9a26268f0999404e8a02f73a5b77259e6ef0bff10c539" Nov 03 21:16:41.491043 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:41.491017 4963 scope.go:122] "RemoveContainer" containerID="882fbddf3e2e325f6bc9a26268f0999404e8a02f73a5b77259e6ef0bff10c539" Nov 03 21:16:41.491468 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:16:41.491444 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"882fbddf3e2e325f6bc9a26268f0999404e8a02f73a5b77259e6ef0bff10c539\": container with ID starting with 882fbddf3e2e325f6bc9a26268f0999404e8a02f73a5b77259e6ef0bff10c539 not found: ID does not exist" containerID="882fbddf3e2e325f6bc9a26268f0999404e8a02f73a5b77259e6ef0bff10c539" Nov 03 21:16:41.491544 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:41.491480 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"882fbddf3e2e325f6bc9a26268f0999404e8a02f73a5b77259e6ef0bff10c539"} err="failed to get container status \"882fbddf3e2e325f6bc9a26268f0999404e8a02f73a5b77259e6ef0bff10c539\": rpc error: code = NotFound desc = could not find container \"882fbddf3e2e325f6bc9a26268f0999404e8a02f73a5b77259e6ef0bff10c539\": container with ID starting with 882fbddf3e2e325f6bc9a26268f0999404e8a02f73a5b77259e6ef0bff10c539 not found: ID does not exist" Nov 03 21:16:41.640461 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:41.640418 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx"] Nov 03 21:16:41.685596 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:41.685553 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-32f6e43c/e2e-lifecycle-model-32f6e43c-kserve-5f97bffb97-tglxx"] Nov 03 21:16:42.529504 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:16:42.529467 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:16:42.969137 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:42.969094 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f315250f-0dc8-4b98-bb9a-9d998e62ac01" path="/var/lib/kubelet/pods/f315250f-0dc8-4b98-bb9a-9d998e62ac01/volumes" Nov 03 21:16:44.411176 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:44.411144 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:48.969034 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:48.968999 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:51.226241 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:51.226202 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:51.417553 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:51.417521 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:53.228261 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:53.228220 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:53.562564 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:53.562465 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:53.754189 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:53.754155 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:55.017848 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:55.017815 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:57.294723 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:57.294684 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:57.513735 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:57.513702 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:57.707918 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:57.707878 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:59.387809 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:59.387776 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:59.935306 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:16:59.935265 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:00.726072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:00.726041 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:00.915570 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:00.915514 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:02.076615 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:02.076576 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:03.124709 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:03.124678 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:04.116615 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:04.116580 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:04.328650 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:04.328616 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:04.858791 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:04.858758 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:05.648611 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:05.648579 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:06.310764 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:06.310734 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:06.517564 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:06.517532 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:07.166209 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:07.166169 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:08.461152 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:08.461113 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:08.649182 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:08.649144 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:09.518401 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:09.518370 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:09.885620 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:09.885544 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:12.449151 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:12.449119 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:12.530547 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:17:12.530510 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:17:12.636320 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:12.636287 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:12.887188 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:12.887095 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:14.483098 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:14.483055 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:14.811324 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:14.811248 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:14.998321 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:14.998276 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:18.328474 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:18.328441 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:18.526797 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:18.526763 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:21.151087 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:21.151048 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:25.691141 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:25.691098 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:30.192644 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:30.192607 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:31.155062 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:31.155022 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:31.174422 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:31.174392 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:17:31.183771 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:31.183745 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:17:34.713837 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:34.713802 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:39.533174 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:39.533133 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:42.530380 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:17:42.530338 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:17:43.111246 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:43.111204 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:43.600301 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:43.600266 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:46.203939 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:46.203873 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:47.217195 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:47.217161 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:47.404428 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:47.404391 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:50.815892 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:50.815858 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:53.398079 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:53.398040 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:55.349852 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.349811 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr"] Nov 03 21:17:55.351770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.351195 4963 cpu_manager.go:414] "containerMap: removing container" podUID="f315250f-0dc8-4b98-bb9a-9d998e62ac01" containerName="main" Nov 03 21:17:55.351770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.351221 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f315250f-0dc8-4b98-bb9a-9d998e62ac01" containerName="main" Nov 03 21:17:55.361275 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.358574 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="f315250f-0dc8-4b98-bb9a-9d998e62ac01" containerName="main" Nov 03 21:17:55.363521 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.363495 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:17:55.408054 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.408021 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr"] Nov 03 21:17:55.451145 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.451108 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b2ad2d-serving-cert\"" Nov 03 21:17:55.498314 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.498273 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr\" (UID: \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:17:55.498600 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.498571 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-metrics-certs\") pod \"maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr\" (UID: \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:17:55.498723 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.498652 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzbrb\" (UniqueName: \"kubernetes.io/projected/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-kube-api-access-hzbrb\") pod \"maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr\" (UID: \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:17:55.498874 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.498851 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr\" (UID: \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:17:55.572277 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.572242 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:55.599828 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.599789 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr\" (UID: \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:17:55.601101 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.599853 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-metrics-certs\") pod \"maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr\" (UID: \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:17:55.601101 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.599941 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzbrb\" (UniqueName: \"kubernetes.io/projected/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-kube-api-access-hzbrb\") pod \"maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr\" (UID: \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:17:55.601101 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.600086 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr\" (UID: \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:17:55.601101 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:17:55.600333 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-serving-cert: secret "maas-api-e2e-ipp-b-b2ad2d-serving-cert" not found Nov 03 21:17:55.601101 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:17:55.600409 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-maas-api-tls podName:0a2b7d0c-a27a-4cd6-8b65-d948191e8af2 nodeName:}" failed. No retries permitted until 2026-11-03 21:17:56.100386659 +0000 UTC m=+1825.260458928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-maas-api-tls") pod "maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" (UID: "0a2b7d0c-a27a-4cd6-8b65-d948191e8af2") : secret "maas-api-e2e-ipp-b-b2ad2d-serving-cert" not found Nov 03 21:17:55.601598 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.601315 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr\" (UID: \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:17:55.603290 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.603258 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-metrics-certs\") pod \"maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr\" (UID: \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:17:55.661346 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.661316 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzbrb\" (UniqueName: \"kubernetes.io/projected/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-kube-api-access-hzbrb\") pod \"maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr\" (UID: \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:17:55.759089 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.759052 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:55.943925 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:55.943870 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:56.107193 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:56.107155 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr\" (UID: \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:17:56.109718 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:56.109688 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr\" (UID: \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:17:56.277955 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:56.277863 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:17:56.655832 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:56.655743 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr"] Nov 03 21:17:56.660543 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:17:56.660501 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2b7d0c_a27a_4cd6_8b65_d948191e8af2.slice/crio-49fad1c530ceed86bb7f8d9e239deb2c392b58052c515c62fe09af00f17a17f1 WatchSource:0}: Error finding container 49fad1c530ceed86bb7f8d9e239deb2c392b58052c515c62fe09af00f17a17f1: Status 404 returned error can't find the container with id 49fad1c530ceed86bb7f8d9e239deb2c392b58052c515c62fe09af00f17a17f1 Nov 03 21:17:57.007663 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:57.007625 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" event={"ID":"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2","Type":"ContainerStarted","Data":"49fad1c530ceed86bb7f8d9e239deb2c392b58052c515c62fe09af00f17a17f1"} Nov 03 21:17:58.015975 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:58.015938 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" event={"ID":"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2","Type":"ContainerStarted","Data":"c5e771a440c08fbe59d7aa69cf4cf306b5c72ef641df2ddd82637d7ce09263bc"} Nov 03 21:17:58.016398 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:58.016058 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:17:58.130039 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:17:58.129974 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" podStartSLOduration=2.10210618 podStartE2EDuration="3.129954191s" totalImagesPullingTime="1.027848011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:17:58.128819855 +0000 UTC m=+1827.288892127" watchObservedRunningTime="2026-11-03 21:17:58.129954191 +0000 UTC m=+1827.290026464" Nov 03 21:18:00.757706 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:00.757676 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:04.027998 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:04.027965 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:18:04.454420 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:04.454301 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:04.871801 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:04.871712 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:07.320096 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:07.320060 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:11.539927 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:11.539827 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr"] Nov 03 21:18:11.544713 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:11.544685 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" Nov 03 21:18:11.600255 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:11.600216 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr"] Nov 03 21:18:11.641760 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:11.641724 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-eae138-serving-cert\"" Nov 03 21:18:11.711114 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:11.711077 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-metrics-certs\") pod \"maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr\" (UID: \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" Nov 03 21:18:11.711320 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:11.711150 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mtbl\" (UniqueName: \"kubernetes.io/projected/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-kube-api-access-4mtbl\") pod \"maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr\" (UID: \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" Nov 03 21:18:11.711320 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:11.711254 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr\" (UID: \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" Nov 03 21:18:11.711452 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:11.711404 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr\" (UID: \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" Nov 03 21:18:11.767692 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:11.767663 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:11.812754 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:11.812666 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-metrics-certs\") pod \"maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr\" (UID: \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" Nov 03 21:18:11.812754 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:11.812750 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mtbl\" (UniqueName: \"kubernetes.io/projected/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-kube-api-access-4mtbl\") pod \"maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr\" (UID: \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" Nov 03 21:18:11.812754 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:11.812795 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr\" (UID: \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" Nov 03 21:18:11.812754 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:11.812953 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr\" (UID: \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" Nov 03 21:18:11.814616 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:11.814581 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr\" (UID: \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" Nov 03 21:18:11.815898 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:11.815870 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr\" (UID: \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" Nov 03 21:18:11.816081 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:11.816005 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-metrics-certs\") pod \"maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr\" (UID: \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" Nov 03 21:18:11.871511 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:11.871470 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mtbl\" (UniqueName: \"kubernetes.io/projected/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-kube-api-access-4mtbl\") pod \"maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr\" (UID: \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" Nov 03 21:18:11.956330 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:11.956293 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:12.160150 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:12.160069 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" Nov 03 21:18:12.211627 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:12.211595 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:12.529213 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:18:12.529175 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:18:12.537109 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:12.537076 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr"] Nov 03 21:18:12.542290 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:18:12.542256 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ddb1ba_594c_4117_b6b2_2e7a06eb90de.slice/crio-e74273be336cd0f8cb937f5d346acb22ad32f4921fd741f9aafb573f5e049c4f WatchSource:0}: Error finding container e74273be336cd0f8cb937f5d346acb22ad32f4921fd741f9aafb573f5e049c4f: Status 404 returned error can't find the container with id e74273be336cd0f8cb937f5d346acb22ad32f4921fd741f9aafb573f5e049c4f Nov 03 21:18:13.119277 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:13.119241 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" event={"ID":"68ddb1ba-594c-4117-b6b2-2e7a06eb90de","Type":"ContainerStarted","Data":"e74273be336cd0f8cb937f5d346acb22ad32f4921fd741f9aafb573f5e049c4f"} Nov 03 21:18:14.127536 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:14.127500 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" event={"ID":"68ddb1ba-594c-4117-b6b2-2e7a06eb90de","Type":"ContainerStarted","Data":"b4f993b5b11c82514e098e00a0d0f0cfe9e7dbd6830f55c584b08ae85e2cbaa2"} Nov 03 21:18:14.127999 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:14.127650 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" Nov 03 21:18:14.241169 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:14.241112 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" podStartSLOduration=2.263353623 podStartE2EDuration="3.241098792s" totalImagesPullingTime="977.745169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:14.239731413 +0000 UTC m=+1843.399803697" watchObservedRunningTime="2026-11-03 21:18:14.241098792 +0000 UTC m=+1843.401171064" Nov 03 21:18:15.977326 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:15.977289 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:16.238928 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:16.238830 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt"] Nov 03 21:18:16.239111 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:16.239052 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt" podUID="d67c6469-1e35-423a-8428-e19f3f7bceb7" containerName="payload-processing" containerID="cri-o://c968a3f9b81f74f2debf8ef31e306895e591d08fac3f3c10c246d9b6aa6d84f0" gracePeriod=30 Nov 03 21:18:16.430592 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:16.430554 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:16.538172 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:16.538143 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt" Nov 03 21:18:16.620899 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:16.620859 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:16.685353 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:16.685301 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d67c6469-1e35-423a-8428-e19f3f7bceb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67c6469-1e35-423a-8428-e19f3f7bceb7-plugins-config-volume\") pod \"d67c6469-1e35-423a-8428-e19f3f7bceb7\" (UID: \"d67c6469-1e35-423a-8428-e19f3f7bceb7\") " Nov 03 21:18:16.685598 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:16.685576 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67c6469-1e35-423a-8428-e19f3f7bceb7-kube-api-access-4mhfd\" (UniqueName: \"kubernetes.io/projected/d67c6469-1e35-423a-8428-e19f3f7bceb7-kube-api-access-4mhfd\") pod \"d67c6469-1e35-423a-8428-e19f3f7bceb7\" (UID: \"d67c6469-1e35-423a-8428-e19f3f7bceb7\") " Nov 03 21:18:16.685704 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:16.685678 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67c6469-1e35-423a-8428-e19f3f7bceb7-plugins-config-volume" pod "d67c6469-1e35-423a-8428-e19f3f7bceb7" (UID: "d67c6469-1e35-423a-8428-e19f3f7bceb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:16.686150 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:16.686127 4963 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67c6469-1e35-423a-8428-e19f3f7bceb7-plugins-config-volume\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:16.687685 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:16.687660 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67c6469-1e35-423a-8428-e19f3f7bceb7-kube-api-access-4mhfd" pod "d67c6469-1e35-423a-8428-e19f3f7bceb7" (UID: "d67c6469-1e35-423a-8428-e19f3f7bceb7"). InnerVolumeSpecName "kube-api-access-4mhfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:16.790469 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:16.787447 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mhfd\" (UniqueName: \"kubernetes.io/projected/d67c6469-1e35-423a-8428-e19f3f7bceb7-kube-api-access-4mhfd\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:17.148885 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:17.148791 4963 generic.go:356] "Generic (PLEG): container finished" podID="d67c6469-1e35-423a-8428-e19f3f7bceb7" containerID="c968a3f9b81f74f2debf8ef31e306895e591d08fac3f3c10c246d9b6aa6d84f0" exitCode=0 Nov 03 21:18:17.148885 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:17.148839 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt" Nov 03 21:18:17.149469 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:17.148878 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt" event={"ID":"d67c6469-1e35-423a-8428-e19f3f7bceb7","Type":"ContainerDied","Data":"c968a3f9b81f74f2debf8ef31e306895e591d08fac3f3c10c246d9b6aa6d84f0"} Nov 03 21:18:17.149469 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:17.148964 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt" event={"ID":"d67c6469-1e35-423a-8428-e19f3f7bceb7","Type":"ContainerDied","Data":"2df33f5ec347c491ab99bdfade524cc8f3d4e6f594593e02ba76456663ebc58c"} Nov 03 21:18:17.149469 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:17.148990 4963 scope.go:122] "RemoveContainer" containerID="c968a3f9b81f74f2debf8ef31e306895e591d08fac3f3c10c246d9b6aa6d84f0" Nov 03 21:18:17.161001 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:17.160977 4963 scope.go:122] "RemoveContainer" containerID="c968a3f9b81f74f2debf8ef31e306895e591d08fac3f3c10c246d9b6aa6d84f0" Nov 03 21:18:17.161296 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:18:17.161276 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c968a3f9b81f74f2debf8ef31e306895e591d08fac3f3c10c246d9b6aa6d84f0\": container with ID starting with c968a3f9b81f74f2debf8ef31e306895e591d08fac3f3c10c246d9b6aa6d84f0 not found: ID does not exist" containerID="c968a3f9b81f74f2debf8ef31e306895e591d08fac3f3c10c246d9b6aa6d84f0" Nov 03 21:18:17.161367 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:17.161305 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c968a3f9b81f74f2debf8ef31e306895e591d08fac3f3c10c246d9b6aa6d84f0"} err="failed to get container status \"c968a3f9b81f74f2debf8ef31e306895e591d08fac3f3c10c246d9b6aa6d84f0\": rpc error: code = NotFound desc = could not find container \"c968a3f9b81f74f2debf8ef31e306895e591d08fac3f3c10c246d9b6aa6d84f0\": container with ID starting with c968a3f9b81f74f2debf8ef31e306895e591d08fac3f3c10c246d9b6aa6d84f0 not found: ID does not exist" Nov 03 21:18:17.306918 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:17.306875 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt"] Nov 03 21:18:17.353553 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:17.353516 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0f8726-7d87bd88fb-cbdwt"] Nov 03 21:18:17.587218 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:17.587182 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:17.784552 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:17.784516 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:18.968018 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:18.967979 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d67c6469-1e35-423a-8428-e19f3f7bceb7" path="/var/lib/kubelet/pods/d67c6469-1e35-423a-8428-e19f3f7bceb7/volumes" Nov 03 21:18:19.166730 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:19.166693 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:19.638143 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:19.638109 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:20.139868 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:20.139835 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" Nov 03 21:18:22.419847 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:22.419812 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:22.605792 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:22.605759 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:23.907976 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:23.907931 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:24.336795 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:24.336763 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:25.010272 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:25.010238 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:27.668094 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:27.668058 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:28.256606 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:28.256572 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:28.450317 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:28.450280 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:28.643296 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:28.643205 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:30.347474 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:30.347434 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:31.311962 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:31.311920 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:31.490427 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:31.490391 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:31.685985 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:31.685823 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:31.871758 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:31.871717 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:32.075652 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:32.075497 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:32.841205 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:32.841173 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:33.923036 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:33.923004 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:37.320792 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:37.320744 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:37.510621 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:37.510583 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:38.769164 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:38.769130 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:39.249049 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:39.249007 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:39.922999 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:39.922963 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:40.412190 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:40.412159 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:40.806787 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:40.806754 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:41.516055 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:41.516022 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:41.727235 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:41.727145 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:42.528727 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:18:42.528687 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:18:45.304467 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:45.304435 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:45.490050 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:45.490018 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:47.370644 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:47.370606 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:49.123758 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:49.123722 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:49.307495 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:49.307463 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:50.225727 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:50.225680 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:54.252245 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:54.252207 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:55.688826 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:55.688788 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:55.870491 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:55.870460 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:56.054540 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:56.054506 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:56.239557 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:56.239523 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:56.421615 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:56.421536 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:56.923200 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:56.923167 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:57.107844 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:57.107807 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:58.568439 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:58.568410 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:58.769179 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:18:58.769149 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:02.046105 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:02.046062 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:02.235053 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:02.235020 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:02.424898 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:02.424816 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:02.902999 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:02.902961 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:07.046387 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:07.046350 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l"] Nov 03 21:19:07.046821 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:07.046598 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" podUID="8e68d020-3255-49f5-b986-b2f07c317894" containerName="istio-proxy" containerID="cri-o://27d12de9ee59adff058c8a103516ff12e10f51322f07cf92ce8510608c6c338d" gracePeriod=30 Nov 03 21:19:07.254543 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:07.254506 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln"] Nov 03 21:19:07.254794 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:07.254767 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" podUID="bafd62fa-db24-4cbf-ad55-1495f7daef2f" containerName="istio-proxy" containerID="cri-o://5f8a0bfa1249a30f6d072da4237f771f08582283a9d258509ac90d89781c33d3" gracePeriod=30 Nov 03 21:19:07.745529 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:07.745499 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:08.481676 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:08.481644 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:12.378826 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.378792 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 21:19:12.379257 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.378856 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" podUID="bafd62fa-db24-4cbf-ad55-1495f7daef2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 21:19:12.391348 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.391316 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:19:12.522494 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.522286 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:19:12.528453 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.528382 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e68d020-3255-49f5-b986-b2f07c317894-istio-token\" (UniqueName: \"kubernetes.io/projected/8e68d020-3255-49f5-b986-b2f07c317894-istio-token\") pod \"8e68d020-3255-49f5-b986-b2f07c317894\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " Nov 03 21:19:12.528453 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.528449 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-workload-certs\") pod \"8e68d020-3255-49f5-b986-b2f07c317894\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " Nov 03 21:19:12.528719 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.528478 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e68d020-3255-49f5-b986-b2f07c317894-kube-api-access-9kprn\" (UniqueName: \"kubernetes.io/projected/8e68d020-3255-49f5-b986-b2f07c317894-kube-api-access-9kprn\") pod \"8e68d020-3255-49f5-b986-b2f07c317894\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " Nov 03 21:19:12.528719 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.528503 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e68d020-3255-49f5-b986-b2f07c317894-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e68d020-3255-49f5-b986-b2f07c317894-istiod-ca-cert\") pod \"8e68d020-3255-49f5-b986-b2f07c317894\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " Nov 03 21:19:12.528719 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.528595 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-istio-envoy\") pod \"8e68d020-3255-49f5-b986-b2f07c317894\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " Nov 03 21:19:12.528719 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.528708 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-workload-socket\") pod \"8e68d020-3255-49f5-b986-b2f07c317894\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " Nov 03 21:19:12.528894 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.528812 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-credential-socket\") pod \"8e68d020-3255-49f5-b986-b2f07c317894\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " Nov 03 21:19:12.528894 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.528816 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-workload-certs" pod "8e68d020-3255-49f5-b986-b2f07c317894" (UID: "8e68d020-3255-49f5-b986-b2f07c317894"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:12.528894 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.528855 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-istio-data\") pod \"8e68d020-3255-49f5-b986-b2f07c317894\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " Nov 03 21:19:12.529052 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.528898 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e68d020-3255-49f5-b986-b2f07c317894-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e68d020-3255-49f5-b986-b2f07c317894-istio-podinfo\") pod \"8e68d020-3255-49f5-b986-b2f07c317894\" (UID: \"8e68d020-3255-49f5-b986-b2f07c317894\") " Nov 03 21:19:12.529104 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.529084 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e68d020-3255-49f5-b986-b2f07c317894-istiod-ca-cert" pod "8e68d020-3255-49f5-b986-b2f07c317894" (UID: "8e68d020-3255-49f5-b986-b2f07c317894"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:12.529153 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.529088 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-workload-socket" pod "8e68d020-3255-49f5-b986-b2f07c317894" (UID: "8e68d020-3255-49f5-b986-b2f07c317894"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:12.529153 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.529133 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-credential-socket" pod "8e68d020-3255-49f5-b986-b2f07c317894" (UID: "8e68d020-3255-49f5-b986-b2f07c317894"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:12.529302 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.529281 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-istio-data" pod "8e68d020-3255-49f5-b986-b2f07c317894" (UID: "8e68d020-3255-49f5-b986-b2f07c317894"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:12.529704 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.529615 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e68d020-3255-49f5-b986-b2f07c317894-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.529704 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.529641 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.529704 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.529657 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.529704 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.529671 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.529704 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.529684 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.531010 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.530982 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-istio-envoy" pod "8e68d020-3255-49f5-b986-b2f07c317894" (UID: "8e68d020-3255-49f5-b986-b2f07c317894"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:12.531201 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.531180 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e68d020-3255-49f5-b986-b2f07c317894-istio-podinfo" pod "8e68d020-3255-49f5-b986-b2f07c317894" (UID: "8e68d020-3255-49f5-b986-b2f07c317894"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:19:12.531260 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.531214 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e68d020-3255-49f5-b986-b2f07c317894-istio-token" pod "8e68d020-3255-49f5-b986-b2f07c317894" (UID: "8e68d020-3255-49f5-b986-b2f07c317894"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:12.531459 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.531445 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e68d020-3255-49f5-b986-b2f07c317894-kube-api-access-9kprn" pod "8e68d020-3255-49f5-b986-b2f07c317894" (UID: "8e68d020-3255-49f5-b986-b2f07c317894"). InnerVolumeSpecName "kube-api-access-9kprn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:12.533038 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:19:12.533012 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:19:12.556984 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.554096 4963 generic.go:356] "Generic (PLEG): container finished" podID="8e68d020-3255-49f5-b986-b2f07c317894" containerID="27d12de9ee59adff058c8a103516ff12e10f51322f07cf92ce8510608c6c338d" exitCode=0 Nov 03 21:19:12.556984 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.554165 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" Nov 03 21:19:12.556984 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.554175 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" event={"ID":"8e68d020-3255-49f5-b986-b2f07c317894","Type":"ContainerDied","Data":"27d12de9ee59adff058c8a103516ff12e10f51322f07cf92ce8510608c6c338d"} Nov 03 21:19:12.556984 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.554234 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l" event={"ID":"8e68d020-3255-49f5-b986-b2f07c317894","Type":"ContainerDied","Data":"5f6e18790ba924914270a1ccf9f8df2197cd1757d650128f4d142884e2797688"} Nov 03 21:19:12.556984 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.554267 4963 scope.go:122] "RemoveContainer" containerID="27d12de9ee59adff058c8a103516ff12e10f51322f07cf92ce8510608c6c338d" Nov 03 21:19:12.560790 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.560764 4963 generic.go:356] "Generic (PLEG): container finished" podID="bafd62fa-db24-4cbf-ad55-1495f7daef2f" containerID="5f8a0bfa1249a30f6d072da4237f771f08582283a9d258509ac90d89781c33d3" exitCode=0 Nov 03 21:19:12.560980 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.560806 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" event={"ID":"bafd62fa-db24-4cbf-ad55-1495f7daef2f","Type":"ContainerDied","Data":"5f8a0bfa1249a30f6d072da4237f771f08582283a9d258509ac90d89781c33d3"} Nov 03 21:19:12.560980 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.560861 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" Nov 03 21:19:12.560980 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.560873 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln" event={"ID":"bafd62fa-db24-4cbf-ad55-1495f7daef2f","Type":"ContainerDied","Data":"0f1feb114da736f44388ebb41486f0d65c15175f9fa96a6613a341e3583f65a4"} Nov 03 21:19:12.573858 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.573830 4963 scope.go:122] "RemoveContainer" containerID="27d12de9ee59adff058c8a103516ff12e10f51322f07cf92ce8510608c6c338d" Nov 03 21:19:12.574225 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:19:12.574205 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d12de9ee59adff058c8a103516ff12e10f51322f07cf92ce8510608c6c338d\": container with ID starting with 27d12de9ee59adff058c8a103516ff12e10f51322f07cf92ce8510608c6c338d not found: ID does not exist" containerID="27d12de9ee59adff058c8a103516ff12e10f51322f07cf92ce8510608c6c338d" Nov 03 21:19:12.574281 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.574236 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d12de9ee59adff058c8a103516ff12e10f51322f07cf92ce8510608c6c338d"} err="failed to get container status \"27d12de9ee59adff058c8a103516ff12e10f51322f07cf92ce8510608c6c338d\": rpc error: code = NotFound desc = could not find container \"27d12de9ee59adff058c8a103516ff12e10f51322f07cf92ce8510608c6c338d\": container with ID starting with 27d12de9ee59adff058c8a103516ff12e10f51322f07cf92ce8510608c6c338d not found: ID does not exist" Nov 03 21:19:12.574281 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.574255 4963 scope.go:122] "RemoveContainer" containerID="5f8a0bfa1249a30f6d072da4237f771f08582283a9d258509ac90d89781c33d3" Nov 03 21:19:12.585213 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.585186 4963 scope.go:122] "RemoveContainer" containerID="5f8a0bfa1249a30f6d072da4237f771f08582283a9d258509ac90d89781c33d3" Nov 03 21:19:12.585592 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:19:12.585560 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f8a0bfa1249a30f6d072da4237f771f08582283a9d258509ac90d89781c33d3\": container with ID starting with 5f8a0bfa1249a30f6d072da4237f771f08582283a9d258509ac90d89781c33d3 not found: ID does not exist" containerID="5f8a0bfa1249a30f6d072da4237f771f08582283a9d258509ac90d89781c33d3" Nov 03 21:19:12.585692 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.585602 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8a0bfa1249a30f6d072da4237f771f08582283a9d258509ac90d89781c33d3"} err="failed to get container status \"5f8a0bfa1249a30f6d072da4237f771f08582283a9d258509ac90d89781c33d3\": rpc error: code = NotFound desc = could not find container \"5f8a0bfa1249a30f6d072da4237f771f08582283a9d258509ac90d89781c33d3\": container with ID starting with 5f8a0bfa1249a30f6d072da4237f771f08582283a9d258509ac90d89781c33d3 not found: ID does not exist" Nov 03 21:19:12.630441 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.630370 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-workload-socket\") pod \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " Nov 03 21:19:12.630441 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.630414 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istiod-ca-cert\") pod \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " Nov 03 21:19:12.630654 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.630489 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-workload-certs\") pod \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " Nov 03 21:19:12.630654 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.630521 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bafd62fa-db24-4cbf-ad55-1495f7daef2f-kube-api-access-9z5bx\" (UniqueName: \"kubernetes.io/projected/bafd62fa-db24-4cbf-ad55-1495f7daef2f-kube-api-access-9z5bx\") pod \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " Nov 03 21:19:12.630654 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.630600 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-token\" (UniqueName: \"kubernetes.io/projected/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-token\") pod \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " Nov 03 21:19:12.630854 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.630664 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-credential-socket\") pod \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " Nov 03 21:19:12.630854 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.630708 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-podinfo\") pod \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " Nov 03 21:19:12.630854 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.630814 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-envoy\") pod \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " Nov 03 21:19:12.631057 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.630864 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-data\") pod \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\" (UID: \"bafd62fa-db24-4cbf-ad55-1495f7daef2f\") " Nov 03 21:19:12.631057 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.630992 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istiod-ca-cert" pod "bafd62fa-db24-4cbf-ad55-1495f7daef2f" (UID: "bafd62fa-db24-4cbf-ad55-1495f7daef2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:12.631530 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.631501 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.631626 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.631535 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e68d020-3255-49f5-b986-b2f07c317894-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.631626 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.631551 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kprn\" (UniqueName: \"kubernetes.io/projected/8e68d020-3255-49f5-b986-b2f07c317894-kube-api-access-9kprn\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.631626 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.631565 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e68d020-3255-49f5-b986-b2f07c317894-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.631626 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.631577 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e68d020-3255-49f5-b986-b2f07c317894-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.631755 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.631655 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-credential-socket" pod "bafd62fa-db24-4cbf-ad55-1495f7daef2f" (UID: "bafd62fa-db24-4cbf-ad55-1495f7daef2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:12.631789 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.631721 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-workload-certs" pod "bafd62fa-db24-4cbf-ad55-1495f7daef2f" (UID: "bafd62fa-db24-4cbf-ad55-1495f7daef2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:12.631828 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.631794 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-data" pod "bafd62fa-db24-4cbf-ad55-1495f7daef2f" (UID: "bafd62fa-db24-4cbf-ad55-1495f7daef2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:12.631861 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.631837 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-workload-socket" pod "bafd62fa-db24-4cbf-ad55-1495f7daef2f" (UID: "bafd62fa-db24-4cbf-ad55-1495f7daef2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:12.633584 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.633561 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-podinfo" pod "bafd62fa-db24-4cbf-ad55-1495f7daef2f" (UID: "bafd62fa-db24-4cbf-ad55-1495f7daef2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:19:12.634144 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.634125 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-token" pod "bafd62fa-db24-4cbf-ad55-1495f7daef2f" (UID: "bafd62fa-db24-4cbf-ad55-1495f7daef2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:12.634261 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.634241 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bafd62fa-db24-4cbf-ad55-1495f7daef2f-kube-api-access-9z5bx" pod "bafd62fa-db24-4cbf-ad55-1495f7daef2f" (UID: "bafd62fa-db24-4cbf-ad55-1495f7daef2f"). InnerVolumeSpecName "kube-api-access-9z5bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:12.634331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.634273 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-envoy" pod "bafd62fa-db24-4cbf-ad55-1495f7daef2f" (UID: "bafd62fa-db24-4cbf-ad55-1495f7daef2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:19:12.651095 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.651057 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:12.732652 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.732616 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.732652 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.732646 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.732652 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.732659 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z5bx\" (UniqueName: \"kubernetes.io/projected/bafd62fa-db24-4cbf-ad55-1495f7daef2f-kube-api-access-9z5bx\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.732943 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.732669 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.732943 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.732678 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.732943 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.732686 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.732943 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.732696 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.732943 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.732704 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafd62fa-db24-4cbf-ad55-1495f7daef2f-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:12.819053 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.819010 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l"] Nov 03 21:19:12.821895 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.821872 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-kh98l"] Nov 03 21:19:12.967534 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:12.967501 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e68d020-3255-49f5-b986-b2f07c317894" path="/var/lib/kubelet/pods/8e68d020-3255-49f5-b986-b2f07c317894/volumes" Nov 03 21:19:13.002217 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:13.002182 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln"] Nov 03 21:19:13.007939 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:13.007872 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-5swln"] Nov 03 21:19:14.968245 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:14.968206 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bafd62fa-db24-4cbf-ad55-1495f7daef2f" path="/var/lib/kubelet/pods/bafd62fa-db24-4cbf-ad55-1495f7daef2f/volumes" Nov 03 21:19:16.437888 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:16.437754 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:16.627809 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:16.627765 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:16.901043 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:16.901008 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:17.125926 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:17.125537 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:17.322928 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:17.317141 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:17.627990 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:17.611345 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:17.802930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:17.802658 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:18.590930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:18.590012 4963 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" podUID="fae009e3-4ccc-46ef-895c-5ecc6ea4e599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:19:22.865262 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:22.865223 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:23.056592 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:23.056536 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:23.256187 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:23.256149 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:23.449472 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:23.449431 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:23.639415 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:23.639330 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:25.372041 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:25.372008 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:27.694034 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:27.693997 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:27.893996 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:27.893964 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:31.834801 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:31.834759 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:32.032989 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:32.032952 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:32.490651 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:32.490603 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:32.674504 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:32.674473 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:33.702962 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:33.702923 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:38.199700 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:38.199658 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:38.394644 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:38.394606 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:38.575952 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:38.575842 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:39.521451 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:39.521400 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:39.708533 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:39.708497 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:42.193913 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:42.193807 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:42.397808 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:42.397750 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:42.529719 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:19:42.529621 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:19:43.078814 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:43.078785 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:44.090567 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:44.090535 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:44.600887 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:44.600850 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:45.428414 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:45.428382 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:48.142824 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:48.142787 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:51.491124 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:51.491087 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:51.749173 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:51.749088 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:52.201168 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:52.201131 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:54.959386 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:19:54.959349 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:01.214548 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:01.214494 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:05.883080 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:05.883038 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:09.233224 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:09.233188 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:09.704530 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:09.704500 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:12.195452 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:12.195409 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:12.531321 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:20:12.531232 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:20:16.711285 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:16.711189 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:18.781726 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:18.781672 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:18.968510 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:18.968478 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:21.066251 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:21.066211 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:21.260163 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:21.260124 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:21.447862 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:21.447830 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:25.755896 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:25.755843 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:30.231492 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:30.231459 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:32.169359 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:32.169326 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:34.998879 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:34.998832 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:36.082806 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:36.082751 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:36.274388 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:36.274314 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:36.544314 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:36.544258 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:36.729956 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:36.729919 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:39.518104 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:39.518063 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:42.528334 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:20:42.528293 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:20:44.020778 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:44.020741 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:48.118706 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.118667 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr"] Nov 03 21:20:48.119192 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.118927 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" podUID="0a2b7d0c-a27a-4cd6-8b65-d948191e8af2" containerName="maas-api" containerID="cri-o://c5e771a440c08fbe59d7aa69cf4cf306b5c72ef641df2ddd82637d7ce09263bc" gracePeriod=30 Nov 03 21:20:48.255952 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.255892 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:48.343164 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.343134 4963 generic.go:356] "Generic (PLEG): container finished" podID="0a2b7d0c-a27a-4cd6-8b65-d948191e8af2" containerID="c5e771a440c08fbe59d7aa69cf4cf306b5c72ef641df2ddd82637d7ce09263bc" exitCode=0 Nov 03 21:20:48.343423 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.343187 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" event={"ID":"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2","Type":"ContainerDied","Data":"c5e771a440c08fbe59d7aa69cf4cf306b5c72ef641df2ddd82637d7ce09263bc"} Nov 03 21:20:48.445233 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.445201 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:20:48.543927 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.540556 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-metrics-certs\") pod \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\" (UID: \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\") " Nov 03 21:20:48.543927 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.540718 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-kube-api-access-hzbrb\" (UniqueName: \"kubernetes.io/projected/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-kube-api-access-hzbrb\") pod \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\" (UID: \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\") " Nov 03 21:20:48.543927 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.540798 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-maas-api-tls\") pod \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\" (UID: \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\") " Nov 03 21:20:48.543927 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.540857 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-openshift-service-ca\") pod \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\" (UID: \"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2\") " Nov 03 21:20:48.543927 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.542126 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-openshift-service-ca" pod "0a2b7d0c-a27a-4cd6-8b65-d948191e8af2" (UID: "0a2b7d0c-a27a-4cd6-8b65-d948191e8af2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:20:48.547736 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.544939 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-metrics-certs" pod "0a2b7d0c-a27a-4cd6-8b65-d948191e8af2" (UID: "0a2b7d0c-a27a-4cd6-8b65-d948191e8af2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:20:48.551097 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.551062 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-maas-api-tls" pod "0a2b7d0c-a27a-4cd6-8b65-d948191e8af2" (UID: "0a2b7d0c-a27a-4cd6-8b65-d948191e8af2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:20:48.553266 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.553231 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-kube-api-access-hzbrb" pod "0a2b7d0c-a27a-4cd6-8b65-d948191e8af2" (UID: "0a2b7d0c-a27a-4cd6-8b65-d948191e8af2"). InnerVolumeSpecName "kube-api-access-hzbrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:20:48.642703 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.642663 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:48.642703 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.642698 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzbrb\" (UniqueName: \"kubernetes.io/projected/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-kube-api-access-hzbrb\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:48.642703 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.642709 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:48.642973 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.642720 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:48.795690 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.795660 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:48.983134 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:48.983103 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:49.351681 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:49.351648 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" Nov 03 21:20:49.351681 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:49.351661 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr" event={"ID":"0a2b7d0c-a27a-4cd6-8b65-d948191e8af2","Type":"ContainerDied","Data":"49fad1c530ceed86bb7f8d9e239deb2c392b58052c515c62fe09af00f17a17f1"} Nov 03 21:20:49.352248 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:49.351733 4963 scope.go:122] "RemoveContainer" containerID="c5e771a440c08fbe59d7aa69cf4cf306b5c72ef641df2ddd82637d7ce09263bc" Nov 03 21:20:49.605681 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:49.603780 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:49.606333 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:49.605886 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr"] Nov 03 21:20:49.612484 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:49.611461 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2ad2d-85c4bb85d-f97vr"] Nov 03 21:20:49.999364 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:49.999333 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:50.389134 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:50.389042 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:50.970501 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:50.970461 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a2b7d0c-a27a-4cd6-8b65-d948191e8af2" path="/var/lib/kubelet/pods/0a2b7d0c-a27a-4cd6-8b65-d948191e8af2/volumes" Nov 03 21:20:51.493985 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:51.493951 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:51.707621 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:51.707590 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:53.002348 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:53.002317 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:53.497778 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:53.497741 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:54.678957 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:54.678887 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:54.902638 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:54.902590 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:55.088845 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:55.088803 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:56.246450 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:56.246417 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:56.857030 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:56.856995 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:57.046497 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:57.046462 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:57.363730 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:57.363681 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:58.274917 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:58.274858 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:58.645890 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:58.645810 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:58.832449 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:20:58.832419 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:00.943640 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:00.943588 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:01.156771 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:01.156714 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:01.842631 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:01.842593 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:02.986214 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:02.986177 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:04.108206 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:04.108174 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:04.305783 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:04.305738 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:04.522295 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:04.522229 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:08.414155 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:08.414104 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:12.529106 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:21:12.529026 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:21:13.554136 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:13.554100 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:18.287214 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:18.287181 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:20.840108 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:20.840053 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:21.024120 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:21.024087 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:23.042766 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:23.042729 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:25.141467 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:25.141425 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:25.355128 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:25.355096 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:26.865016 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:26.864984 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:27.443924 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:27.406513 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:30.097050 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:30.097003 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:32.516126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:32.516097 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:34.756767 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:34.756733 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:39.311169 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:39.311133 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:42.528713 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:21:42.528674 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:21:43.965560 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:43.965525 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:45.481293 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:45.481265 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:48.733095 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:48.733051 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:51.506361 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:51.506326 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:55.012787 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:55.012750 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:55.215728 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:55.215686 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:55.401437 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:55.401339 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:59.742042 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:21:59.742005 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:03.900304 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:03.900265 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4"] Nov 03 21:22:03.900870 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:03.900855 4963 cpu_manager.go:414] "containerMap: removing container" podUID="0a2b7d0c-a27a-4cd6-8b65-d948191e8af2" containerName="maas-api" Nov 03 21:22:03.900992 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:03.900871 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2b7d0c-a27a-4cd6-8b65-d948191e8af2" containerName="maas-api" Nov 03 21:22:03.900992 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:03.900896 4963 cpu_manager.go:414] "containerMap: removing container" podUID="d67c6469-1e35-423a-8428-e19f3f7bceb7" containerName="payload-processing" Nov 03 21:22:03.900992 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:03.900921 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d67c6469-1e35-423a-8428-e19f3f7bceb7" containerName="payload-processing" Nov 03 21:22:03.900992 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:03.900942 4963 cpu_manager.go:414] "containerMap: removing container" podUID="bafd62fa-db24-4cbf-ad55-1495f7daef2f" containerName="istio-proxy" Nov 03 21:22:03.900992 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:03.900949 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bafd62fa-db24-4cbf-ad55-1495f7daef2f" containerName="istio-proxy" Nov 03 21:22:03.900992 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:03.900969 4963 cpu_manager.go:414] "containerMap: removing container" podUID="8e68d020-3255-49f5-b986-b2f07c317894" containerName="istio-proxy" Nov 03 21:22:03.900992 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:03.900977 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e68d020-3255-49f5-b986-b2f07c317894" containerName="istio-proxy" Nov 03 21:22:03.901232 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:03.901106 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="d67c6469-1e35-423a-8428-e19f3f7bceb7" containerName="payload-processing" Nov 03 21:22:03.901232 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:03.901125 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="bafd62fa-db24-4cbf-ad55-1495f7daef2f" containerName="istio-proxy" Nov 03 21:22:03.901232 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:03.901138 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e68d020-3255-49f5-b986-b2f07c317894" containerName="istio-proxy" Nov 03 21:22:03.901232 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:03.901150 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a2b7d0c-a27a-4cd6-8b65-d948191e8af2" containerName="maas-api" Nov 03 21:22:03.904877 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:03.904855 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:22:03.956221 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:03.956188 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-1190cc-serving-cert\"" Nov 03 21:22:04.006611 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:04.006521 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4"] Nov 03 21:22:04.034340 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:04.034303 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgxkh\" (UniqueName: \"kubernetes.io/projected/035dbca4-b720-4fc5-84a6-5b3e421c20d5-kube-api-access-jgxkh\") pod \"maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4\" (UID: \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:22:04.034621 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:04.034593 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/035dbca4-b720-4fc5-84a6-5b3e421c20d5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4\" (UID: \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:22:04.034846 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:04.034823 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/035dbca4-b720-4fc5-84a6-5b3e421c20d5-metrics-certs\") pod \"maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4\" (UID: \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:22:04.035308 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:04.034872 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/035dbca4-b720-4fc5-84a6-5b3e421c20d5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4\" (UID: \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:22:04.128801 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:04.128766 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:04.136766 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:04.136729 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/035dbca4-b720-4fc5-84a6-5b3e421c20d5-metrics-certs\") pod \"maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4\" (UID: \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:22:04.136893 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:04.136787 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/035dbca4-b720-4fc5-84a6-5b3e421c20d5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4\" (UID: \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:22:04.136965 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:04.136943 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgxkh\" (UniqueName: \"kubernetes.io/projected/035dbca4-b720-4fc5-84a6-5b3e421c20d5-kube-api-access-jgxkh\") pod \"maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4\" (UID: \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:22:04.137004 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:04.136980 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/035dbca4-b720-4fc5-84a6-5b3e421c20d5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4\" (UID: \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:22:04.137169 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:22:04.137155 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-serving-cert: secret "maas-api-e2e-praxis-a-1190cc-serving-cert" not found Nov 03 21:22:04.137250 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:22:04.137233 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/035dbca4-b720-4fc5-84a6-5b3e421c20d5-maas-api-tls podName:035dbca4-b720-4fc5-84a6-5b3e421c20d5 nodeName:}" failed. No retries permitted until 2026-11-03 21:22:04.6372124 +0000 UTC m=+2073.797284656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/035dbca4-b720-4fc5-84a6-5b3e421c20d5-maas-api-tls") pod "maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" (UID: "035dbca4-b720-4fc5-84a6-5b3e421c20d5") : secret "maas-api-e2e-praxis-a-1190cc-serving-cert" not found Nov 03 21:22:04.137569 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:04.137545 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/035dbca4-b720-4fc5-84a6-5b3e421c20d5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4\" (UID: \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:22:04.139813 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:04.139782 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/035dbca4-b720-4fc5-84a6-5b3e421c20d5-metrics-certs\") pod \"maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4\" (UID: \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:22:04.196646 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:04.196617 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgxkh\" (UniqueName: \"kubernetes.io/projected/035dbca4-b720-4fc5-84a6-5b3e421c20d5-kube-api-access-jgxkh\") pod \"maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4\" (UID: \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:22:04.316846 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:04.316810 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:04.555450 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:04.555362 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:04.643551 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:04.643513 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/035dbca4-b720-4fc5-84a6-5b3e421c20d5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4\" (UID: \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:22:04.646113 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:04.646087 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/035dbca4-b720-4fc5-84a6-5b3e421c20d5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4\" (UID: \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:22:04.819346 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:04.819249 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:22:05.207750 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:05.207683 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4"] Nov 03 21:22:05.212594 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:22:05.212567 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod035dbca4_b720_4fc5_84a6_5b3e421c20d5.slice/crio-9d7fed3cb873e162ba0ae43f5d810a7fbc07102c995d683b7b9299ce9c04415b WatchSource:0}: Error finding container 9d7fed3cb873e162ba0ae43f5d810a7fbc07102c995d683b7b9299ce9c04415b: Status 404 returned error can't find the container with id 9d7fed3cb873e162ba0ae43f5d810a7fbc07102c995d683b7b9299ce9c04415b Nov 03 21:22:05.213876 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:05.213855 4963 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:22:05.896928 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:05.896862 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" event={"ID":"035dbca4-b720-4fc5-84a6-5b3e421c20d5","Type":"ContainerStarted","Data":"9d7fed3cb873e162ba0ae43f5d810a7fbc07102c995d683b7b9299ce9c04415b"} Nov 03 21:22:06.904131 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:06.904094 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" event={"ID":"035dbca4-b720-4fc5-84a6-5b3e421c20d5","Type":"ContainerStarted","Data":"9da46f0bcd1dce5eee10032e67a232701037e10326978214f06bef0d7dd85bc1"} Nov 03 21:22:06.904567 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:06.904244 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:22:07.017987 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:07.017836 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" podStartSLOduration=3.011692742 podStartE2EDuration="4.01781501s" totalImagesPullingTime="1.006122268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:07.016994903 +0000 UTC m=+2076.177067176" watchObservedRunningTime="2026-11-03 21:22:07.01781501 +0000 UTC m=+2076.177887285" Nov 03 21:22:09.188811 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:09.188771 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:12.528724 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:22:12.528680 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:22:12.926979 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:12.926770 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:22:15.307844 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:15.307803 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:20.012770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:20.012733 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:23.559470 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:23.559431 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9"] Nov 03 21:22:23.560157 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:23.559640 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" podUID="8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" containerName="istio-proxy" containerID="cri-o://71bc0214c392dc67537c95b57804834f25de33c8c6d3761482b4ed444b279806" gracePeriod=30 Nov 03 21:22:23.799333 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:23.799298 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw"] Nov 03 21:22:23.799580 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:23.799553 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" podUID="3c2b9bb1-4418-46a4-941d-def176133161" containerName="istio-proxy" containerID="cri-o://de89f7eecfeeac97967c1a7164d13e58bd95c5c9b08bb8481110316153107962" gracePeriod=30 Nov 03 21:22:24.711561 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:24.711511 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:28.073443 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:28.073410 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:28.555063 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:28.555034 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:28.576678 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:28.576314 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:22:28.576678 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:28.576382 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" podUID="3c2b9bb1-4418-46a4-941d-def176133161" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:22:28.954723 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:28.954691 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:22:29.048398 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.048365 4963 generic.go:356] "Generic (PLEG): container finished" podID="8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" containerID="71bc0214c392dc67537c95b57804834f25de33c8c6d3761482b4ed444b279806" exitCode=0 Nov 03 21:22:29.048591 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.048452 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" event={"ID":"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0","Type":"ContainerDied","Data":"71bc0214c392dc67537c95b57804834f25de33c8c6d3761482b4ed444b279806"} Nov 03 21:22:29.048591 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.048543 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" event={"ID":"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0","Type":"ContainerDied","Data":"47ca428e051b33e79b93b4053058701d6e50ae17a1d10d896ca2a7352547099c"} Nov 03 21:22:29.048726 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.048596 4963 scope.go:122] "RemoveContainer" containerID="71bc0214c392dc67537c95b57804834f25de33c8c6d3761482b4ed444b279806" Nov 03 21:22:29.048857 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.048836 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9" Nov 03 21:22:29.050477 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.050455 4963 generic.go:356] "Generic (PLEG): container finished" podID="3c2b9bb1-4418-46a4-941d-def176133161" containerID="de89f7eecfeeac97967c1a7164d13e58bd95c5c9b08bb8481110316153107962" exitCode=0 Nov 03 21:22:29.050605 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.050580 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" event={"ID":"3c2b9bb1-4418-46a4-941d-def176133161","Type":"ContainerDied","Data":"de89f7eecfeeac97967c1a7164d13e58bd95c5c9b08bb8481110316153107962"} Nov 03 21:22:29.057153 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.057122 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-workload-certs\") pod \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " Nov 03 21:22:29.057303 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.057174 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-data\") pod \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " Nov 03 21:22:29.057303 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.057219 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-envoy\") pod \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " Nov 03 21:22:29.057303 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.057248 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-podinfo\") pod \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " Nov 03 21:22:29.057665 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.057314 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-workload-socket\") pod \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " Nov 03 21:22:29.057665 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.057384 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-token\" (UniqueName: \"kubernetes.io/projected/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-token\") pod \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " Nov 03 21:22:29.057665 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.057418 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istiod-ca-cert\") pod \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " Nov 03 21:22:29.057665 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.057503 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-workload-certs" pod "8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" (UID: "8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:22:29.057665 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.057598 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-kube-api-access-jn7zm\" (UniqueName: \"kubernetes.io/projected/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-kube-api-access-jn7zm\") pod \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " Nov 03 21:22:29.057665 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.057627 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-credential-socket\") pod \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\" (UID: \"8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0\") " Nov 03 21:22:29.058551 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.058057 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-data" pod "8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" (UID: "8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:22:29.058551 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.058374 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:29.058551 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.058397 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:29.058757 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.058573 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-workload-socket" pod "8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" (UID: "8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:22:29.059415 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.059383 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istiod-ca-cert" pod "8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" (UID: "8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:22:29.059922 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.059882 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-credential-socket" pod "8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" (UID: "8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:22:29.062059 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.062033 4963 scope.go:122] "RemoveContainer" containerID="71bc0214c392dc67537c95b57804834f25de33c8c6d3761482b4ed444b279806" Nov 03 21:22:29.062428 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:22:29.062394 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71bc0214c392dc67537c95b57804834f25de33c8c6d3761482b4ed444b279806\": container with ID starting with 71bc0214c392dc67537c95b57804834f25de33c8c6d3761482b4ed444b279806 not found: ID does not exist" containerID="71bc0214c392dc67537c95b57804834f25de33c8c6d3761482b4ed444b279806" Nov 03 21:22:29.062504 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.062436 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71bc0214c392dc67537c95b57804834f25de33c8c6d3761482b4ed444b279806"} err="failed to get container status \"71bc0214c392dc67537c95b57804834f25de33c8c6d3761482b4ed444b279806\": rpc error: code = NotFound desc = could not find container \"71bc0214c392dc67537c95b57804834f25de33c8c6d3761482b4ed444b279806\": container with ID starting with 71bc0214c392dc67537c95b57804834f25de33c8c6d3761482b4ed444b279806 not found: ID does not exist" Nov 03 21:22:29.064053 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.064031 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-envoy" pod "8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" (UID: "8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:22:29.065660 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.065630 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-podinfo" pod "8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" (UID: "8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:22:29.065744 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.065695 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-token" pod "8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" (UID: "8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:22:29.065744 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.065716 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-kube-api-access-jn7zm" pod "8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" (UID: "8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0"). InnerVolumeSpecName "kube-api-access-jn7zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:22:29.096242 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.096215 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:22:29.158819 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.158779 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c2b9bb1-4418-46a4-941d-def176133161-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2b9bb1-4418-46a4-941d-def176133161-istio-podinfo\") pod \"3c2b9bb1-4418-46a4-941d-def176133161\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " Nov 03 21:22:29.159051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.158872 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-istio-envoy\") pod \"3c2b9bb1-4418-46a4-941d-def176133161\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " Nov 03 21:22:29.159051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.158928 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c2b9bb1-4418-46a4-941d-def176133161-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2b9bb1-4418-46a4-941d-def176133161-istiod-ca-cert\") pod \"3c2b9bb1-4418-46a4-941d-def176133161\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " Nov 03 21:22:29.159051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.158969 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2b9bb1-4418-46a4-941d-def176133161-kube-api-access-s5t6q\" (UniqueName: \"kubernetes.io/projected/3c2b9bb1-4418-46a4-941d-def176133161-kube-api-access-s5t6q\") pod \"3c2b9bb1-4418-46a4-941d-def176133161\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " Nov 03 21:22:29.159051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.159023 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-credential-socket\") pod \"3c2b9bb1-4418-46a4-941d-def176133161\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " Nov 03 21:22:29.159318 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.159177 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-istio-data\") pod \"3c2b9bb1-4418-46a4-941d-def176133161\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " Nov 03 21:22:29.159318 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.159226 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-workload-socket\") pod \"3c2b9bb1-4418-46a4-941d-def176133161\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " Nov 03 21:22:29.159461 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.159324 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-workload-certs\") pod \"3c2b9bb1-4418-46a4-941d-def176133161\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " Nov 03 21:22:29.159461 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.159363 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2b9bb1-4418-46a4-941d-def176133161-istio-token\" (UniqueName: \"kubernetes.io/projected/3c2b9bb1-4418-46a4-941d-def176133161-istio-token\") pod \"3c2b9bb1-4418-46a4-941d-def176133161\" (UID: \"3c2b9bb1-4418-46a4-941d-def176133161\") " Nov 03 21:22:29.159890 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.159589 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-istio-data" pod "3c2b9bb1-4418-46a4-941d-def176133161" (UID: "3c2b9bb1-4418-46a4-941d-def176133161"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:22:29.159890 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.159654 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c2b9bb1-4418-46a4-941d-def176133161-istiod-ca-cert" pod "3c2b9bb1-4418-46a4-941d-def176133161" (UID: "3c2b9bb1-4418-46a4-941d-def176133161"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:22:29.159890 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.159844 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-workload-certs" pod "3c2b9bb1-4418-46a4-941d-def176133161" (UID: "3c2b9bb1-4418-46a4-941d-def176133161"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:22:29.159890 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.159875 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-credential-socket" pod "3c2b9bb1-4418-46a4-941d-def176133161" (UID: "3c2b9bb1-4418-46a4-941d-def176133161"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:22:29.160185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.160063 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2b9bb1-4418-46a4-941d-def176133161-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:29.160185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.160085 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:29.160185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.160100 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn7zm\" (UniqueName: \"kubernetes.io/projected/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-kube-api-access-jn7zm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:29.160185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.160117 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:29.160185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.160130 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:29.160185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.160145 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:29.160185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.160159 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:29.160185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.160173 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:29.160185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.160186 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:29.160519 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.160200 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:29.160519 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.160213 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:29.160770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.160750 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-workload-socket" pod "3c2b9bb1-4418-46a4-941d-def176133161" (UID: "3c2b9bb1-4418-46a4-941d-def176133161"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:22:29.161675 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.161655 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c2b9bb1-4418-46a4-941d-def176133161-istio-podinfo" pod "3c2b9bb1-4418-46a4-941d-def176133161" (UID: "3c2b9bb1-4418-46a4-941d-def176133161"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:22:29.161968 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.161943 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-istio-envoy" pod "3c2b9bb1-4418-46a4-941d-def176133161" (UID: "3c2b9bb1-4418-46a4-941d-def176133161"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:22:29.162064 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.162002 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2b9bb1-4418-46a4-941d-def176133161-kube-api-access-s5t6q" pod "3c2b9bb1-4418-46a4-941d-def176133161" (UID: "3c2b9bb1-4418-46a4-941d-def176133161"). InnerVolumeSpecName "kube-api-access-s5t6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:22:29.162185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.162170 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2b9bb1-4418-46a4-941d-def176133161-istio-token" pod "3c2b9bb1-4418-46a4-941d-def176133161" (UID: "3c2b9bb1-4418-46a4-941d-def176133161"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:22:29.261024 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.260974 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:29.261024 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.261010 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2b9bb1-4418-46a4-941d-def176133161-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:29.261024 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.261024 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2b9bb1-4418-46a4-941d-def176133161-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:29.261278 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.261035 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2b9bb1-4418-46a4-941d-def176133161-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:29.261278 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.261048 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5t6q\" (UniqueName: \"kubernetes.io/projected/3c2b9bb1-4418-46a4-941d-def176133161-kube-api-access-s5t6q\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:29.466621 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.466576 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9"] Nov 03 21:22:29.470514 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.470486 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-jrps9"] Nov 03 21:22:29.832838 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:29.832798 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:30.058610 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:30.058577 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" Nov 03 21:22:30.058610 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:30.058590 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw" event={"ID":"3c2b9bb1-4418-46a4-941d-def176133161","Type":"ContainerDied","Data":"a694089071deadff135697f4c7d4d07e8683dc6cb2ec49a4d8050dfb3d79bf82"} Nov 03 21:22:30.058864 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:30.058656 4963 scope.go:122] "RemoveContainer" containerID="de89f7eecfeeac97967c1a7164d13e58bd95c5c9b08bb8481110316153107962" Nov 03 21:22:30.220913 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:30.220877 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw"] Nov 03 21:22:30.227667 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:30.227637 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9zqfw"] Nov 03 21:22:30.466226 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:30.466021 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:30.841647 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:30.841616 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:30.969032 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:30.969000 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c2b9bb1-4418-46a4-941d-def176133161" path="/var/lib/kubelet/pods/3c2b9bb1-4418-46a4-941d-def176133161/volumes" Nov 03 21:22:30.969535 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:30.969521 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" path="/var/lib/kubelet/pods/8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0/volumes" Nov 03 21:22:31.196703 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:31.196662 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:31.226526 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:31.226479 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:22:31.235809 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:31.235777 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:22:32.823569 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:32.823534 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:34.610238 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:34.610205 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:34.809107 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:34.809071 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:35.169460 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:35.169406 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:36.711089 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:36.710865 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:37.990512 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:37.990483 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:38.317427 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.317334 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:38.482337 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.482300 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z"] Nov 03 21:22:38.482932 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.482891 4963 cpu_manager.go:414] "containerMap: removing container" podUID="8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" containerName="istio-proxy" Nov 03 21:22:38.482932 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.482920 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" containerName="istio-proxy" Nov 03 21:22:38.483021 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.482964 4963 cpu_manager.go:414] "containerMap: removing container" podUID="3c2b9bb1-4418-46a4-941d-def176133161" containerName="istio-proxy" Nov 03 21:22:38.483021 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.482970 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c2b9bb1-4418-46a4-941d-def176133161" containerName="istio-proxy" Nov 03 21:22:38.483081 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.483053 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c2b9bb1-4418-46a4-941d-def176133161" containerName="istio-proxy" Nov 03 21:22:38.483081 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.483064 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8b4ff4-9263-4e71-9e1e-c0bbd1a30ee0" containerName="istio-proxy" Nov 03 21:22:38.486476 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.486451 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" Nov 03 21:22:38.489550 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.489514 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4b1b6e-bc5a-4854-97df-e14361e3f643-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z\" (UID: \"db4b1b6e-bc5a-4854-97df-e14361e3f643\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" Nov 03 21:22:38.489669 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.489588 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57t65\" (UniqueName: \"kubernetes.io/projected/db4b1b6e-bc5a-4854-97df-e14361e3f643-kube-api-access-57t65\") pod \"payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z\" (UID: \"db4b1b6e-bc5a-4854-97df-e14361e3f643\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" Nov 03 21:22:38.512590 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.512564 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:38.541529 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.541495 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z"] Nov 03 21:22:38.587978 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.587224 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3d317d\"" Nov 03 21:22:38.587978 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.587422 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3d317d-dockercfg-rqfj2\"" Nov 03 21:22:38.591711 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.591655 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4b1b6e-bc5a-4854-97df-e14361e3f643-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z\" (UID: \"db4b1b6e-bc5a-4854-97df-e14361e3f643\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" Nov 03 21:22:38.591966 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.591754 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57t65\" (UniqueName: \"kubernetes.io/projected/db4b1b6e-bc5a-4854-97df-e14361e3f643-kube-api-access-57t65\") pod \"payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z\" (UID: \"db4b1b6e-bc5a-4854-97df-e14361e3f643\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" Nov 03 21:22:38.593040 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.593000 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4b1b6e-bc5a-4854-97df-e14361e3f643-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z\" (UID: \"db4b1b6e-bc5a-4854-97df-e14361e3f643\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" Nov 03 21:22:38.594496 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.594462 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm"] Nov 03 21:22:38.614225 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.614188 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" Nov 03 21:22:38.682054 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.682008 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57t65\" (UniqueName: \"kubernetes.io/projected/db4b1b6e-bc5a-4854-97df-e14361e3f643-kube-api-access-57t65\") pod \"payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z\" (UID: \"db4b1b6e-bc5a-4854-97df-e14361e3f643\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" Nov 03 21:22:38.693386 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.693347 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jfmq\" (UniqueName: \"kubernetes.io/projected/6e9e38ae-d8ba-402e-bda3-312acd372dec-kube-api-access-9jfmq\") pod \"payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm\" (UID: \"6e9e38ae-d8ba-402e-bda3-312acd372dec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" Nov 03 21:22:38.693619 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.693545 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9e38ae-d8ba-402e-bda3-312acd372dec-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm\" (UID: \"6e9e38ae-d8ba-402e-bda3-312acd372dec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" Nov 03 21:22:38.694083 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.694064 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm"] Nov 03 21:22:38.794249 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.794207 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9e38ae-d8ba-402e-bda3-312acd372dec-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm\" (UID: \"6e9e38ae-d8ba-402e-bda3-312acd372dec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" Nov 03 21:22:38.794460 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.794322 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jfmq\" (UniqueName: \"kubernetes.io/projected/6e9e38ae-d8ba-402e-bda3-312acd372dec-kube-api-access-9jfmq\") pod \"payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm\" (UID: \"6e9e38ae-d8ba-402e-bda3-312acd372dec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" Nov 03 21:22:38.794953 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.794890 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9e38ae-d8ba-402e-bda3-312acd372dec-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm\" (UID: \"6e9e38ae-d8ba-402e-bda3-312acd372dec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" Nov 03 21:22:38.808738 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.808709 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" Nov 03 21:22:38.898936 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.897021 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jfmq\" (UniqueName: \"kubernetes.io/projected/6e9e38ae-d8ba-402e-bda3-312acd372dec-kube-api-access-9jfmq\") pod \"payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm\" (UID: \"6e9e38ae-d8ba-402e-bda3-312acd372dec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" Nov 03 21:22:38.997168 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:38.997137 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" Nov 03 21:22:39.196352 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:39.196305 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z"] Nov 03 21:22:39.202784 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:22:39.202749 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4b1b6e_bc5a_4854_97df_e14361e3f643.slice/crio-4c5b3147717678f318d8bee754dfff5c5c4a79e4c05fd657b8351ff3042357e2 WatchSource:0}: Error finding container 4c5b3147717678f318d8bee754dfff5c5c4a79e4c05fd657b8351ff3042357e2: Status 404 returned error can't find the container with id 4c5b3147717678f318d8bee754dfff5c5c4a79e4c05fd657b8351ff3042357e2 Nov 03 21:22:39.376567 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:39.376532 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm"] Nov 03 21:22:39.382043 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:22:39.382017 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9e38ae_d8ba_402e_bda3_312acd372dec.slice/crio-3c2a9f4e2e72bee5e9abf9edd55d6d01153cbd5cb35e9ab8044b38be23741d35 WatchSource:0}: Error finding container 3c2a9f4e2e72bee5e9abf9edd55d6d01153cbd5cb35e9ab8044b38be23741d35: Status 404 returned error can't find the container with id 3c2a9f4e2e72bee5e9abf9edd55d6d01153cbd5cb35e9ab8044b38be23741d35 Nov 03 21:22:40.128892 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:40.128799 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" event={"ID":"6e9e38ae-d8ba-402e-bda3-312acd372dec","Type":"ContainerStarted","Data":"3c2a9f4e2e72bee5e9abf9edd55d6d01153cbd5cb35e9ab8044b38be23741d35"} Nov 03 21:22:40.130261 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:40.130234 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" event={"ID":"db4b1b6e-bc5a-4854-97df-e14361e3f643","Type":"ContainerStarted","Data":"4c5b3147717678f318d8bee754dfff5c5c4a79e4c05fd657b8351ff3042357e2"} Nov 03 21:22:42.577113 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:22:42.577075 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:22:42.942455 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:42.942420 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:43.155955 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:43.155919 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" event={"ID":"6e9e38ae-d8ba-402e-bda3-312acd372dec","Type":"ContainerStarted","Data":"b54cccbfcc8b6a579ea160b8e55f2c82fad7d9cdb010e2a6a393a2ed5927208b"} Nov 03 21:22:43.156155 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:43.156003 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" Nov 03 21:22:43.157789 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:43.157751 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" event={"ID":"db4b1b6e-bc5a-4854-97df-e14361e3f643","Type":"ContainerStarted","Data":"e765734b2f7d9550911de5a44355baa7dad05251ea33d50ec52ad2ae421da081"} Nov 03 21:22:43.157955 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:43.157816 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" Nov 03 21:22:43.272872 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:43.272767 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" podStartSLOduration=2.080197598 podStartE2EDuration="5.272752189s" totalImagesPullingTime="3.192554591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:43.269751799 +0000 UTC m=+2112.429824071" watchObservedRunningTime="2026-11-03 21:22:43.272752189 +0000 UTC m=+2112.432824485" Nov 03 21:22:43.380113 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:43.380046 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" podStartSLOduration=2.152801221 podStartE2EDuration="5.38003061s" totalImagesPullingTime="3.227229389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:43.379059437 +0000 UTC m=+2112.539131734" watchObservedRunningTime="2026-11-03 21:22:43.38003061 +0000 UTC m=+2112.540102925" Nov 03 21:22:47.538424 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:47.538387 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:51.213092 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.213043 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg"] Nov 03 21:22:51.223008 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.222978 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.276327 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.276290 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg"] Nov 03 21:22:51.320868 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.320828 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-eae138\"/\"openshift-service-ca.crt\"" Nov 03 21:22:51.320868 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.320845 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-eae138\"/\"kube-root-ca.crt\"" Nov 03 21:22:51.321153 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.320947 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-eae138\"/\"default-dockercfg-pmsfp\"" Nov 03 21:22:51.321153 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.321029 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-eae138\"/\"ipp-route-eae138-kserve-self-signed-certs\"" Nov 03 21:22:51.354110 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.354067 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:51.374723 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.374676 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-dshm\") pod \"ipp-route-eae138-kserve-779bdf84d5-ffhxg\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.375226 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.375183 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/295db5cb-7753-4ca5-aecb-cbb3a1851a87-tls-certs\") pod \"ipp-route-eae138-kserve-779bdf84d5-ffhxg\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.375663 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.375636 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5sjc\" (UniqueName: \"kubernetes.io/projected/295db5cb-7753-4ca5-aecb-cbb3a1851a87-kube-api-access-m5sjc\") pod \"ipp-route-eae138-kserve-779bdf84d5-ffhxg\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.375830 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.375711 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-tmp-dir\") pod \"ipp-route-eae138-kserve-779bdf84d5-ffhxg\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.375830 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.375747 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-model-cache\") pod \"ipp-route-eae138-kserve-779bdf84d5-ffhxg\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.375830 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.375812 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-home\") pod \"ipp-route-eae138-kserve-779bdf84d5-ffhxg\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.477774 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.477677 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/295db5cb-7753-4ca5-aecb-cbb3a1851a87-tls-certs\") pod \"ipp-route-eae138-kserve-779bdf84d5-ffhxg\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.477977 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.477846 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5sjc\" (UniqueName: \"kubernetes.io/projected/295db5cb-7753-4ca5-aecb-cbb3a1851a87-kube-api-access-m5sjc\") pod \"ipp-route-eae138-kserve-779bdf84d5-ffhxg\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.477977 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.477891 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-tmp-dir\") pod \"ipp-route-eae138-kserve-779bdf84d5-ffhxg\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.477977 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.477966 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-model-cache\") pod \"ipp-route-eae138-kserve-779bdf84d5-ffhxg\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.478160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.478010 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-home\") pod \"ipp-route-eae138-kserve-779bdf84d5-ffhxg\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.478160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.478066 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-dshm\") pod \"ipp-route-eae138-kserve-779bdf84d5-ffhxg\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.478377 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.478340 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-tmp-dir\") pod \"ipp-route-eae138-kserve-779bdf84d5-ffhxg\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.478540 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.478514 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-model-cache\") pod \"ipp-route-eae138-kserve-779bdf84d5-ffhxg\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.478540 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.478542 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-home\") pod \"ipp-route-eae138-kserve-779bdf84d5-ffhxg\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.480846 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.480815 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-dshm\") pod \"ipp-route-eae138-kserve-779bdf84d5-ffhxg\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.481072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.481050 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/295db5cb-7753-4ca5-aecb-cbb3a1851a87-tls-certs\") pod \"ipp-route-eae138-kserve-779bdf84d5-ffhxg\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.537285 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.537246 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5sjc\" (UniqueName: \"kubernetes.io/projected/295db5cb-7753-4ca5-aecb-cbb3a1851a87-kube-api-access-m5sjc\") pod \"ipp-route-eae138-kserve-779bdf84d5-ffhxg\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:51.835399 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:51.835291 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:52.126528 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:52.126451 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:52.221812 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:52.221777 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg"] Nov 03 21:22:52.227230 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:22:52.227193 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod295db5cb_7753_4ca5_aecb_cbb3a1851a87.slice/crio-a8ca3f5120e717ca8c604b514682a7f4f201847d3be1d7ff697ecacf7720fca2 WatchSource:0}: Error finding container a8ca3f5120e717ca8c604b514682a7f4f201847d3be1d7ff697ecacf7720fca2: Status 404 returned error can't find the container with id a8ca3f5120e717ca8c604b514682a7f4f201847d3be1d7ff697ecacf7720fca2 Nov 03 21:22:52.422068 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:52.421988 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:53.223140 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:53.223096 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" event={"ID":"295db5cb-7753-4ca5-aecb-cbb3a1851a87","Type":"ContainerStarted","Data":"f75a4870a1f8ce41a449839ac4840638524a2362f1641e1619710c60528e2b0d"} Nov 03 21:22:53.223610 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:53.223151 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" event={"ID":"295db5cb-7753-4ca5-aecb-cbb3a1851a87","Type":"ContainerStarted","Data":"a8ca3f5120e717ca8c604b514682a7f4f201847d3be1d7ff697ecacf7720fca2"} Nov 03 21:22:53.337098 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:53.337028 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" podStartSLOduration=2.337008386 podStartE2EDuration="2.337008386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:53.333859742 +0000 UTC m=+2122.493932016" watchObservedRunningTime="2026-11-03 21:22:53.337008386 +0000 UTC m=+2122.497080660" Nov 03 21:22:53.835806 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:53.835764 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:22:54.167942 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:54.167842 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" Nov 03 21:22:54.167942 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:54.167923 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" Nov 03 21:22:56.097608 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:56.097575 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:56.573815 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:56.573783 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:56.761242 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:56.761201 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:57.790373 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:22:57.790336 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:01.790193 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:01.790158 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:01.835404 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:01.835367 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:23:01.852268 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:01.852236 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:23:01.977596 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:01.977556 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:02.291788 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:02.291757 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:23:02.364317 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:02.364286 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:03.621887 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:03.621858 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:04.116497 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:04.116468 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:05.006274 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:05.006240 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:07.170835 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:07.170797 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:10.049559 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:10.049521 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:11.292800 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:11.292761 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:12.002025 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:12.001977 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:12.529071 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:23:12.529036 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:23:15.206181 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:15.206129 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:15.841748 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:15.841716 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:16.050246 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:16.050209 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:16.421125 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:16.421094 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:17.707881 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:17.707840 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:17.899629 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:17.899598 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:18.804143 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:18.804108 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:19.787484 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:19.787446 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:19.983172 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:19.983133 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:20.167291 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:20.167213 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:21.878996 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:21.878965 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:22.406843 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:22.406810 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:23.587857 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:23.587824 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:23.788186 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:23.788152 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:28.884632 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:28.884601 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:33.165196 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:33.165148 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:34.529518 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:34.529485 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:35.004257 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:35.004226 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:35.272093 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:35.272012 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:36.152604 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:36.152577 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:36.334016 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:36.333986 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:36.524195 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:36.524162 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:40.782567 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:40.782537 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:42.527521 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:23:42.527480 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:23:44.212787 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:44.212746 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:44.400373 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:44.400329 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:45.348812 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:45.348773 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:45.808583 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:45.808551 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:46.333620 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:46.333574 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:47.934363 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:47.934329 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:48.600948 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:48.600607 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:48.831086 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:48.831049 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:49.673249 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:49.673195 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:51.449432 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:51.449397 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:51.638177 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:51.638147 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:52.535829 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:52.535791 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:52.727891 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:52.727859 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:54.490975 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:54.490937 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822"] Nov 03 21:23:54.491350 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:54.491148 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" podUID="faeed472-64ed-40e2-aee1-0703462ccbe1" containerName="istio-proxy" containerID="cri-o://f44154bae9ca2ef65a260cb15365c81b744d8605a802c0ce6e42c133dc762d14" gracePeriod=30 Nov 03 21:23:55.246256 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:55.246220 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:55.443147 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:55.443116 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:55.657468 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:55.657387 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:56.436838 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:56.436804 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:56.620182 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:56.620143 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:57.359301 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:57.359244 4963 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-s5822 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:23:57.359737 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:57.359314 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" podUID="faeed472-64ed-40e2-aee1-0703462ccbe1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:23:57.584801 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:57.584769 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:58.117092 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:58.117059 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:58.894394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:58.894364 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:59.663307 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.663261 4963 generic.go:356] "Generic (PLEG): container finished" podID="faeed472-64ed-40e2-aee1-0703462ccbe1" containerID="f44154bae9ca2ef65a260cb15365c81b744d8605a802c0ce6e42c133dc762d14" exitCode=0 Nov 03 21:23:59.663513 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.663326 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" event={"ID":"faeed472-64ed-40e2-aee1-0703462ccbe1","Type":"ContainerDied","Data":"f44154bae9ca2ef65a260cb15365c81b744d8605a802c0ce6e42c133dc762d14"} Nov 03 21:23:59.767721 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.767656 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:23:59.866925 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.866874 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-credential-socket\") pod \"faeed472-64ed-40e2-aee1-0703462ccbe1\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " Nov 03 21:23:59.867107 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.866938 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-workload-certs\") pod \"faeed472-64ed-40e2-aee1-0703462ccbe1\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " Nov 03 21:23:59.867107 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.866970 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-data\") pod \"faeed472-64ed-40e2-aee1-0703462ccbe1\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " Nov 03 21:23:59.867214 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.867137 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-podinfo\") pod \"faeed472-64ed-40e2-aee1-0703462ccbe1\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " Nov 03 21:23:59.867214 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.867168 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-envoy\") pod \"faeed472-64ed-40e2-aee1-0703462ccbe1\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " Nov 03 21:23:59.867214 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.867197 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-token\" (UniqueName: \"kubernetes.io/projected/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-token\") pod \"faeed472-64ed-40e2-aee1-0703462ccbe1\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " Nov 03 21:23:59.867370 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.867219 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-credential-socket" pod "faeed472-64ed-40e2-aee1-0703462ccbe1" (UID: "faeed472-64ed-40e2-aee1-0703462ccbe1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:23:59.867370 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.867260 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faeed472-64ed-40e2-aee1-0703462ccbe1-kube-api-access-ddbfn\" (UniqueName: \"kubernetes.io/projected/faeed472-64ed-40e2-aee1-0703462ccbe1-kube-api-access-ddbfn\") pod \"faeed472-64ed-40e2-aee1-0703462ccbe1\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " Nov 03 21:23:59.867370 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.867261 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-workload-certs" pod "faeed472-64ed-40e2-aee1-0703462ccbe1" (UID: "faeed472-64ed-40e2-aee1-0703462ccbe1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:23:59.867370 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.867311 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faeed472-64ed-40e2-aee1-0703462ccbe1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faeed472-64ed-40e2-aee1-0703462ccbe1-istiod-ca-cert\") pod \"faeed472-64ed-40e2-aee1-0703462ccbe1\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " Nov 03 21:23:59.867370 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.867364 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-data" pod "faeed472-64ed-40e2-aee1-0703462ccbe1" (UID: "faeed472-64ed-40e2-aee1-0703462ccbe1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:23:59.867622 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.867369 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-workload-socket\") pod \"faeed472-64ed-40e2-aee1-0703462ccbe1\" (UID: \"faeed472-64ed-40e2-aee1-0703462ccbe1\") " Nov 03 21:23:59.867622 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.867596 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-workload-socket" pod "faeed472-64ed-40e2-aee1-0703462ccbe1" (UID: "faeed472-64ed-40e2-aee1-0703462ccbe1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:23:59.867920 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.867876 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faeed472-64ed-40e2-aee1-0703462ccbe1-istiod-ca-cert" pod "faeed472-64ed-40e2-aee1-0703462ccbe1" (UID: "faeed472-64ed-40e2-aee1-0703462ccbe1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:23:59.868053 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.868038 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:59.868114 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.868060 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:59.868114 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.868076 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:59.868114 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.868089 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faeed472-64ed-40e2-aee1-0703462ccbe1-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:59.868114 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.868104 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:59.869622 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.869599 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-token" pod "faeed472-64ed-40e2-aee1-0703462ccbe1" (UID: "faeed472-64ed-40e2-aee1-0703462ccbe1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:23:59.869895 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.869866 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-envoy" pod "faeed472-64ed-40e2-aee1-0703462ccbe1" (UID: "faeed472-64ed-40e2-aee1-0703462ccbe1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:23:59.870023 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.869897 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-podinfo" pod "faeed472-64ed-40e2-aee1-0703462ccbe1" (UID: "faeed472-64ed-40e2-aee1-0703462ccbe1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:23:59.870103 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.870086 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faeed472-64ed-40e2-aee1-0703462ccbe1-kube-api-access-ddbfn" pod "faeed472-64ed-40e2-aee1-0703462ccbe1" (UID: "faeed472-64ed-40e2-aee1-0703462ccbe1"). InnerVolumeSpecName "kube-api-access-ddbfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:23:59.969434 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.969396 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:59.969434 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.969436 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:59.970051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.969459 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faeed472-64ed-40e2-aee1-0703462ccbe1-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:59.970051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:23:59.969475 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddbfn\" (UniqueName: \"kubernetes.io/projected/faeed472-64ed-40e2-aee1-0703462ccbe1-kube-api-access-ddbfn\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:00.670891 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:00.670856 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" Nov 03 21:24:00.670891 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:00.670892 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822" event={"ID":"faeed472-64ed-40e2-aee1-0703462ccbe1","Type":"ContainerDied","Data":"c9d49b0efb494815a39d1c4623a707afdcfb78c7f507d907004985a04255b01d"} Nov 03 21:24:00.671151 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:00.670975 4963 scope.go:122] "RemoveContainer" containerID="f44154bae9ca2ef65a260cb15365c81b744d8605a802c0ce6e42c133dc762d14" Nov 03 21:24:00.828944 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:00.828883 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822"] Nov 03 21:24:00.832987 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:00.832963 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-s5822"] Nov 03 21:24:00.971969 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:00.971929 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faeed472-64ed-40e2-aee1-0703462ccbe1" path="/var/lib/kubelet/pods/faeed472-64ed-40e2-aee1-0703462ccbe1/volumes" Nov 03 21:24:01.402426 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:01.402334 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:05.019069 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:05.019030 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:05.769049 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:05.769003 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:05.971793 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:05.971594 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:06.404042 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:06.404011 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:10.994100 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:10.994019 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:11.347177 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:11.347090 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:12.529152 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:24:12.529112 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:24:17.796666 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:17.796628 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:17.983359 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:17.983325 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:22.402899 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:22.402860 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:26.968026 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:26.967989 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:31.659767 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:31.659728 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:33.505507 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:33.505467 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:37.715959 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:37.715893 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:42.223414 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:42.223357 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:42.528660 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:24:42.528566 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:24:47.004441 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:47.004402 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:48.945494 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:48.945462 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:49.766018 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:49.765984 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:50.054884 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:50.054804 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:51.877259 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:51.877223 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:52.145673 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:52.145594 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:52.331827 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:52.331794 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:54.740879 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:54.740842 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:54.930442 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:54.930405 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:56.799104 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:56.799064 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:56.984183 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:24:56.984141 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:00.114099 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:00.114046 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:00.295883 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:00.295850 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:01.175166 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:01.175128 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:02.792821 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:02.792785 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:07.256602 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:07.256561 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:10.879058 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:10.879025 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:11.115770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:11.115738 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:11.901451 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:11.901414 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:12.387332 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:12.387297 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:12.529236 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:25:12.529189 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:25:13.017824 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:13.017788 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:18.313750 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:18.313708 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:22.803680 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:22.803644 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:27.606139 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:27.606099 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:29.823482 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:29.823445 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:32.297584 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:32.297546 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:33.853482 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:33.853437 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:35.992583 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:35.992549 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:37.320579 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:37.320543 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:37.508084 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:37.508046 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:38.501861 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:38.501828 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:40.569345 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:40.569258 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:40.751225 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:40.751196 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:41.784235 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:41.784175 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:42.528978 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:25:42.528939 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:25:43.112376 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:43.112346 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:45.521857 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:45.521824 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:47.647759 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:47.647717 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:50.443324 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:50.443290 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg"] Nov 03 21:25:50.558252 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:50.558210 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:52.144351 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:52.144304 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:56.563839 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:56.563783 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:56.749155 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:56.749124 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:56.934471 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:25:56.934389 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:01.563450 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:01.563411 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:01.751016 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:01.750979 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:02.356786 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:02.356750 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:02.653231 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:02.653136 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:05.465072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.465021 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" podUID="295db5cb-7753-4ca5-aecb-cbb3a1851a87" containerName="main" containerID="cri-o://f75a4870a1f8ce41a449839ac4840638524a2362f1641e1619710c60528e2b0d" gracePeriod=45 Nov 03 21:26:05.746295 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.746261 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:26:05.880793 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.880759 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295db5cb-7753-4ca5-aecb-cbb3a1851a87-kube-api-access-m5sjc\" (UniqueName: \"kubernetes.io/projected/295db5cb-7753-4ca5-aecb-cbb3a1851a87-kube-api-access-m5sjc\") pod \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " Nov 03 21:26:05.881000 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.880808 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-tmp-dir\") pod \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " Nov 03 21:26:05.881000 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.880967 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/295db5cb-7753-4ca5-aecb-cbb3a1851a87-tls-certs\" (UniqueName: \"kubernetes.io/secret/295db5cb-7753-4ca5-aecb-cbb3a1851a87-tls-certs\") pod \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " Nov 03 21:26:05.881122 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.881017 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-dshm\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-dshm\") pod \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " Nov 03 21:26:05.881181 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.881128 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-tmp-dir" pod "295db5cb-7753-4ca5-aecb-cbb3a1851a87" (UID: "295db5cb-7753-4ca5-aecb-cbb3a1851a87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:05.881244 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.881180 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-model-cache\") pod \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " Nov 03 21:26:05.881315 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.881283 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-home\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-home\") pod \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\" (UID: \"295db5cb-7753-4ca5-aecb-cbb3a1851a87\") " Nov 03 21:26:05.881452 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.881425 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-model-cache" pod "295db5cb-7753-4ca5-aecb-cbb3a1851a87" (UID: "295db5cb-7753-4ca5-aecb-cbb3a1851a87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:05.881634 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.881606 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-home" pod "295db5cb-7753-4ca5-aecb-cbb3a1851a87" (UID: "295db5cb-7753-4ca5-aecb-cbb3a1851a87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:05.881854 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.881832 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:05.881981 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.881859 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:05.881981 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.881875 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:05.883550 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.883513 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-dshm" pod "295db5cb-7753-4ca5-aecb-cbb3a1851a87" (UID: "295db5cb-7753-4ca5-aecb-cbb3a1851a87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:26:05.883654 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.883549 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/295db5cb-7753-4ca5-aecb-cbb3a1851a87-tls-certs" pod "295db5cb-7753-4ca5-aecb-cbb3a1851a87" (UID: "295db5cb-7753-4ca5-aecb-cbb3a1851a87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:26:05.883654 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.883552 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295db5cb-7753-4ca5-aecb-cbb3a1851a87-kube-api-access-m5sjc" pod "295db5cb-7753-4ca5-aecb-cbb3a1851a87" (UID: "295db5cb-7753-4ca5-aecb-cbb3a1851a87"). InnerVolumeSpecName "kube-api-access-m5sjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:05.983247 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.983166 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/295db5cb-7753-4ca5-aecb-cbb3a1851a87-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:05.983247 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.983194 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/295db5cb-7753-4ca5-aecb-cbb3a1851a87-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:05.983247 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:05.983211 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5sjc\" (UniqueName: \"kubernetes.io/projected/295db5cb-7753-4ca5-aecb-cbb3a1851a87-kube-api-access-m5sjc\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:06.229365 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:06.229331 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:06.419549 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:06.419469 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:06.508674 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:06.508637 4963 generic.go:356] "Generic (PLEG): container finished" podID="295db5cb-7753-4ca5-aecb-cbb3a1851a87" containerID="f75a4870a1f8ce41a449839ac4840638524a2362f1641e1619710c60528e2b0d" exitCode=0 Nov 03 21:26:06.509251 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:06.508702 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" Nov 03 21:26:06.509251 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:06.508700 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" event={"ID":"295db5cb-7753-4ca5-aecb-cbb3a1851a87","Type":"ContainerDied","Data":"f75a4870a1f8ce41a449839ac4840638524a2362f1641e1619710c60528e2b0d"} Nov 03 21:26:06.509251 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:06.508822 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg" event={"ID":"295db5cb-7753-4ca5-aecb-cbb3a1851a87","Type":"ContainerDied","Data":"a8ca3f5120e717ca8c604b514682a7f4f201847d3be1d7ff697ecacf7720fca2"} Nov 03 21:26:06.509251 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:06.508850 4963 scope.go:122] "RemoveContainer" containerID="f75a4870a1f8ce41a449839ac4840638524a2362f1641e1619710c60528e2b0d" Nov 03 21:26:06.522642 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:06.522615 4963 scope.go:122] "RemoveContainer" containerID="f75a4870a1f8ce41a449839ac4840638524a2362f1641e1619710c60528e2b0d" Nov 03 21:26:06.523025 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:26:06.523002 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f75a4870a1f8ce41a449839ac4840638524a2362f1641e1619710c60528e2b0d\": container with ID starting with f75a4870a1f8ce41a449839ac4840638524a2362f1641e1619710c60528e2b0d not found: ID does not exist" containerID="f75a4870a1f8ce41a449839ac4840638524a2362f1641e1619710c60528e2b0d" Nov 03 21:26:06.523103 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:06.523036 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75a4870a1f8ce41a449839ac4840638524a2362f1641e1619710c60528e2b0d"} err="failed to get container status \"f75a4870a1f8ce41a449839ac4840638524a2362f1641e1619710c60528e2b0d\": rpc error: code = NotFound desc = could not find container \"f75a4870a1f8ce41a449839ac4840638524a2362f1641e1619710c60528e2b0d\": container with ID starting with f75a4870a1f8ce41a449839ac4840638524a2362f1641e1619710c60528e2b0d not found: ID does not exist" Nov 03 21:26:06.666488 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:06.666451 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg"] Nov 03 21:26:06.714679 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:06.714643 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-eae138/ipp-route-eae138-kserve-779bdf84d5-ffhxg"] Nov 03 21:26:06.970435 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:06.970354 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="295db5cb-7753-4ca5-aecb-cbb3a1851a87" path="/var/lib/kubelet/pods/295db5cb-7753-4ca5-aecb-cbb3a1851a87/volumes" Nov 03 21:26:08.293363 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:08.293314 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:09.473609 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:09.473573 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:09.661019 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:09.660985 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:10.508840 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:10.508801 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4"] Nov 03 21:26:10.509359 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:10.509067 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" podUID="035dbca4-b720-4fc5-84a6-5b3e421c20d5" containerName="maas-api" containerID="cri-o://9da46f0bcd1dce5eee10032e67a232701037e10326978214f06bef0d7dd85bc1" gracePeriod=30 Nov 03 21:26:10.609604 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:10.609562 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:10.821415 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:10.821378 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:26:10.944865 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:10.944829 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/035dbca4-b720-4fc5-84a6-5b3e421c20d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/035dbca4-b720-4fc5-84a6-5b3e421c20d5-maas-api-tls\") pod \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\" (UID: \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\") " Nov 03 21:26:10.945071 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:10.944891 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/035dbca4-b720-4fc5-84a6-5b3e421c20d5-kube-api-access-jgxkh\" (UniqueName: \"kubernetes.io/projected/035dbca4-b720-4fc5-84a6-5b3e421c20d5-kube-api-access-jgxkh\") pod \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\" (UID: \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\") " Nov 03 21:26:10.945071 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:10.944965 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/035dbca4-b720-4fc5-84a6-5b3e421c20d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/035dbca4-b720-4fc5-84a6-5b3e421c20d5-metrics-certs\") pod \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\" (UID: \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\") " Nov 03 21:26:10.945182 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:10.945103 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/035dbca4-b720-4fc5-84a6-5b3e421c20d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/035dbca4-b720-4fc5-84a6-5b3e421c20d5-openshift-service-ca\") pod \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\" (UID: \"035dbca4-b720-4fc5-84a6-5b3e421c20d5\") " Nov 03 21:26:10.945948 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:10.945888 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/035dbca4-b720-4fc5-84a6-5b3e421c20d5-openshift-service-ca" pod "035dbca4-b720-4fc5-84a6-5b3e421c20d5" (UID: "035dbca4-b720-4fc5-84a6-5b3e421c20d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:26:10.947899 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:10.947848 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/035dbca4-b720-4fc5-84a6-5b3e421c20d5-kube-api-access-jgxkh" pod "035dbca4-b720-4fc5-84a6-5b3e421c20d5" (UID: "035dbca4-b720-4fc5-84a6-5b3e421c20d5"). InnerVolumeSpecName "kube-api-access-jgxkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:10.947899 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:10.947848 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/035dbca4-b720-4fc5-84a6-5b3e421c20d5-maas-api-tls" pod "035dbca4-b720-4fc5-84a6-5b3e421c20d5" (UID: "035dbca4-b720-4fc5-84a6-5b3e421c20d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:26:10.948269 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:10.948244 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/035dbca4-b720-4fc5-84a6-5b3e421c20d5-metrics-certs" pod "035dbca4-b720-4fc5-84a6-5b3e421c20d5" (UID: "035dbca4-b720-4fc5-84a6-5b3e421c20d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:26:11.046345 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:11.046253 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/035dbca4-b720-4fc5-84a6-5b3e421c20d5-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:11.046345 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:11.046287 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgxkh\" (UniqueName: \"kubernetes.io/projected/035dbca4-b720-4fc5-84a6-5b3e421c20d5-kube-api-access-jgxkh\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:11.046345 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:11.046303 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/035dbca4-b720-4fc5-84a6-5b3e421c20d5-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:11.046345 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:11.046314 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/035dbca4-b720-4fc5-84a6-5b3e421c20d5-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:11.548850 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:11.548762 4963 generic.go:356] "Generic (PLEG): container finished" podID="035dbca4-b720-4fc5-84a6-5b3e421c20d5" containerID="9da46f0bcd1dce5eee10032e67a232701037e10326978214f06bef0d7dd85bc1" exitCode=0 Nov 03 21:26:11.548850 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:11.548843 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" Nov 03 21:26:11.549449 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:11.549014 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" event={"ID":"035dbca4-b720-4fc5-84a6-5b3e421c20d5","Type":"ContainerDied","Data":"9da46f0bcd1dce5eee10032e67a232701037e10326978214f06bef0d7dd85bc1"} Nov 03 21:26:11.549449 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:11.549072 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4" event={"ID":"035dbca4-b720-4fc5-84a6-5b3e421c20d5","Type":"ContainerDied","Data":"9d7fed3cb873e162ba0ae43f5d810a7fbc07102c995d683b7b9299ce9c04415b"} Nov 03 21:26:11.549449 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:11.549104 4963 scope.go:122] "RemoveContainer" containerID="9da46f0bcd1dce5eee10032e67a232701037e10326978214f06bef0d7dd85bc1" Nov 03 21:26:11.559827 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:11.559805 4963 scope.go:122] "RemoveContainer" containerID="9da46f0bcd1dce5eee10032e67a232701037e10326978214f06bef0d7dd85bc1" Nov 03 21:26:11.560148 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:26:11.560123 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9da46f0bcd1dce5eee10032e67a232701037e10326978214f06bef0d7dd85bc1\": container with ID starting with 9da46f0bcd1dce5eee10032e67a232701037e10326978214f06bef0d7dd85bc1 not found: ID does not exist" containerID="9da46f0bcd1dce5eee10032e67a232701037e10326978214f06bef0d7dd85bc1" Nov 03 21:26:11.560212 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:11.560158 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da46f0bcd1dce5eee10032e67a232701037e10326978214f06bef0d7dd85bc1"} err="failed to get container status \"9da46f0bcd1dce5eee10032e67a232701037e10326978214f06bef0d7dd85bc1\": rpc error: code = NotFound desc = could not find container \"9da46f0bcd1dce5eee10032e67a232701037e10326978214f06bef0d7dd85bc1\": container with ID starting with 9da46f0bcd1dce5eee10032e67a232701037e10326978214f06bef0d7dd85bc1 not found: ID does not exist" Nov 03 21:26:11.707855 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:11.707821 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4"] Nov 03 21:26:11.755855 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:11.755820 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1190cc-65cbb7d69c-xljs4"] Nov 03 21:26:12.528625 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:26:12.528589 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:26:12.622552 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:12.622510 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:12.969122 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:12.969090 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="035dbca4-b720-4fc5-84a6-5b3e421c20d5" path="/var/lib/kubelet/pods/035dbca4-b720-4fc5-84a6-5b3e421c20d5/volumes" Nov 03 21:26:14.460075 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:14.460038 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:14.651177 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:14.651139 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:15.993371 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:15.993304 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:16.514065 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:16.514015 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:17.330898 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:17.330862 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:19.054211 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:19.054172 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:23.797952 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:23.797849 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:24.353281 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:24.353237 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:24.901880 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:24.901842 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:25.086803 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:25.086763 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:26.756487 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:26.756446 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:28.412566 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:28.412534 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:28.807571 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:28.807536 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:30.218617 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:30.218579 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:33.805308 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:33.805262 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:34.191889 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:34.191863 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:34.372789 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:34.372725 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:39.959835 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:39.959804 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:40.492974 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:40.492932 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:42.528824 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:26:42.528786 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:26:44.526651 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:44.526610 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:48.985538 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:48.985500 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:53.523573 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:53.523535 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:58.015519 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:26:58.015486 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:02.627029 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:02.626971 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:07.266225 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:07.266192 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:07.446628 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:07.446597 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:07.633345 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:07.633254 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:12.120015 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:12.119944 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:12.529676 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:27:12.529638 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:27:16.144270 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:16.144231 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:18.007810 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:18.007777 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:20.774530 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:20.774502 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:20.956751 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:20.956714 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:22.418252 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:22.418210 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:22.850117 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:22.850081 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:23.034588 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:23.034554 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:27.664851 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:27.664812 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:27.865670 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:27.865639 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:31.268728 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:31.268698 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:27:31.282054 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:31.282027 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:27:33.848183 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:33.848148 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:34.518628 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:34.518595 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:38.347841 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:38.347730 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:39.807149 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:39.807107 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:42.532316 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:27:42.532274 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:27:42.818297 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:42.818223 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:47.326327 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:47.326293 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:51.825502 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:51.825461 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:56.408640 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:56.408601 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:56.447585 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:56.447547 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll"] Nov 03 21:27:56.447862 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:27:56.447835 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" podUID="30c75fe2-5f87-4852-9f1f-703f4ff01304" containerName="istio-proxy" containerID="cri-o://a667b2f37c93fa0ef914966a11a326318626447b55621b9e0f5ecb599850ce65" gracePeriod=30 Nov 03 21:28:00.924699 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:00.924664 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:01.714855 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.714782 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:28:01.814113 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.814077 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-envoy\") pod \"30c75fe2-5f87-4852-9f1f-703f4ff01304\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " Nov 03 21:28:01.814296 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.814168 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-workload-certs\") pod \"30c75fe2-5f87-4852-9f1f-703f4ff01304\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " Nov 03 21:28:01.814296 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.814211 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-workload-socket\") pod \"30c75fe2-5f87-4852-9f1f-703f4ff01304\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " Nov 03 21:28:01.814296 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.814278 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c75fe2-5f87-4852-9f1f-703f4ff01304-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c75fe2-5f87-4852-9f1f-703f4ff01304-istiod-ca-cert\") pod \"30c75fe2-5f87-4852-9f1f-703f4ff01304\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " Nov 03 21:28:01.814437 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.814313 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-credential-socket\") pod \"30c75fe2-5f87-4852-9f1f-703f4ff01304\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " Nov 03 21:28:01.814437 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.814407 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c75fe2-5f87-4852-9f1f-703f4ff01304-kube-api-access-vdvj9\" (UniqueName: \"kubernetes.io/projected/30c75fe2-5f87-4852-9f1f-703f4ff01304-kube-api-access-vdvj9\") pod \"30c75fe2-5f87-4852-9f1f-703f4ff01304\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " Nov 03 21:28:01.814534 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.814479 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-data\") pod \"30c75fe2-5f87-4852-9f1f-703f4ff01304\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " Nov 03 21:28:01.814534 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.814492 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-workload-certs" pod "30c75fe2-5f87-4852-9f1f-703f4ff01304" (UID: "30c75fe2-5f87-4852-9f1f-703f4ff01304"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:01.814534 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.814529 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-token\" (UniqueName: \"kubernetes.io/projected/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-token\") pod \"30c75fe2-5f87-4852-9f1f-703f4ff01304\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " Nov 03 21:28:01.814687 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.814574 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-workload-socket" pod "30c75fe2-5f87-4852-9f1f-703f4ff01304" (UID: "30c75fe2-5f87-4852-9f1f-703f4ff01304"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:01.814687 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.814636 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-podinfo\") pod \"30c75fe2-5f87-4852-9f1f-703f4ff01304\" (UID: \"30c75fe2-5f87-4852-9f1f-703f4ff01304\") " Nov 03 21:28:01.814788 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.814689 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c75fe2-5f87-4852-9f1f-703f4ff01304-istiod-ca-cert" pod "30c75fe2-5f87-4852-9f1f-703f4ff01304" (UID: "30c75fe2-5f87-4852-9f1f-703f4ff01304"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:28:01.814788 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.814693 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-credential-socket" pod "30c75fe2-5f87-4852-9f1f-703f4ff01304" (UID: "30c75fe2-5f87-4852-9f1f-703f4ff01304"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:01.814895 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.814791 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-data" pod "30c75fe2-5f87-4852-9f1f-703f4ff01304" (UID: "30c75fe2-5f87-4852-9f1f-703f4ff01304"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:01.815126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.815100 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:01.815324 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.815123 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:01.815324 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.815146 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c75fe2-5f87-4852-9f1f-703f4ff01304-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:01.815324 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.815157 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:01.815324 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.815166 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:01.816638 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.816611 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-envoy" pod "30c75fe2-5f87-4852-9f1f-703f4ff01304" (UID: "30c75fe2-5f87-4852-9f1f-703f4ff01304"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:28:01.816751 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.816724 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-token" pod "30c75fe2-5f87-4852-9f1f-703f4ff01304" (UID: "30c75fe2-5f87-4852-9f1f-703f4ff01304"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:01.816819 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.816759 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c75fe2-5f87-4852-9f1f-703f4ff01304-kube-api-access-vdvj9" pod "30c75fe2-5f87-4852-9f1f-703f4ff01304" (UID: "30c75fe2-5f87-4852-9f1f-703f4ff01304"). InnerVolumeSpecName "kube-api-access-vdvj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:01.816819 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.816792 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-podinfo" pod "30c75fe2-5f87-4852-9f1f-703f4ff01304" (UID: "30c75fe2-5f87-4852-9f1f-703f4ff01304"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:28:01.916220 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.916180 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:01.916220 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.916214 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:01.916220 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.916223 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c75fe2-5f87-4852-9f1f-703f4ff01304-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:01.916544 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:01.916233 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdvj9\" (UniqueName: \"kubernetes.io/projected/30c75fe2-5f87-4852-9f1f-703f4ff01304-kube-api-access-vdvj9\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:02.303912 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:02.303864 4963 generic.go:356] "Generic (PLEG): container finished" podID="30c75fe2-5f87-4852-9f1f-703f4ff01304" containerID="a667b2f37c93fa0ef914966a11a326318626447b55621b9e0f5ecb599850ce65" exitCode=0 Nov 03 21:28:02.304330 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:02.303953 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" Nov 03 21:28:02.304330 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:02.303954 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" event={"ID":"30c75fe2-5f87-4852-9f1f-703f4ff01304","Type":"ContainerDied","Data":"a667b2f37c93fa0ef914966a11a326318626447b55621b9e0f5ecb599850ce65"} Nov 03 21:28:02.304330 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:02.304013 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll" event={"ID":"30c75fe2-5f87-4852-9f1f-703f4ff01304","Type":"ContainerDied","Data":"a42392eae3b54db16e3547aeac7b0e585bada8bb6a6716cdacb7a79b8d3df7c7"} Nov 03 21:28:02.304330 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:02.304038 4963 scope.go:122] "RemoveContainer" containerID="a667b2f37c93fa0ef914966a11a326318626447b55621b9e0f5ecb599850ce65" Nov 03 21:28:02.314843 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:02.314822 4963 scope.go:122] "RemoveContainer" containerID="a667b2f37c93fa0ef914966a11a326318626447b55621b9e0f5ecb599850ce65" Nov 03 21:28:02.315153 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:28:02.315133 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a667b2f37c93fa0ef914966a11a326318626447b55621b9e0f5ecb599850ce65\": container with ID starting with a667b2f37c93fa0ef914966a11a326318626447b55621b9e0f5ecb599850ce65 not found: ID does not exist" containerID="a667b2f37c93fa0ef914966a11a326318626447b55621b9e0f5ecb599850ce65" Nov 03 21:28:02.315211 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:02.315164 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a667b2f37c93fa0ef914966a11a326318626447b55621b9e0f5ecb599850ce65"} err="failed to get container status \"a667b2f37c93fa0ef914966a11a326318626447b55621b9e0f5ecb599850ce65\": rpc error: code = NotFound desc = could not find container \"a667b2f37c93fa0ef914966a11a326318626447b55621b9e0f5ecb599850ce65\": container with ID starting with a667b2f37c93fa0ef914966a11a326318626447b55621b9e0f5ecb599850ce65 not found: ID does not exist" Nov 03 21:28:02.462187 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:02.462153 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll"] Nov 03 21:28:02.466435 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:02.466403 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-pj4ll"] Nov 03 21:28:02.968787 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:02.968754 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c75fe2-5f87-4852-9f1f-703f4ff01304" path="/var/lib/kubelet/pods/30c75fe2-5f87-4852-9f1f-703f4ff01304/volumes" Nov 03 21:28:05.468643 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:05.468613 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:09.338762 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:09.338727 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:09.429694 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:09.429659 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr"] Nov 03 21:28:09.429951 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:09.429922 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" podUID="68ddb1ba-594c-4117-b6b2-2e7a06eb90de" containerName="maas-api" containerID="cri-o://b4f993b5b11c82514e098e00a0d0f0cfe9e7dbd6830f55c584b08ae85e2cbaa2" gracePeriod=30 Nov 03 21:28:09.578629 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:09.578554 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:09.737946 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:09.737887 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" Nov 03 21:28:09.804126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:09.804091 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-metrics-certs\") pod \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\" (UID: \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\") " Nov 03 21:28:09.804340 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:09.804143 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-kube-api-access-4mtbl\" (UniqueName: \"kubernetes.io/projected/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-kube-api-access-4mtbl\") pod \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\" (UID: \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\") " Nov 03 21:28:09.804340 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:09.804227 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-openshift-service-ca\") pod \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\" (UID: \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\") " Nov 03 21:28:09.804464 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:09.804354 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-maas-api-tls\") pod \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\" (UID: \"68ddb1ba-594c-4117-b6b2-2e7a06eb90de\") " Nov 03 21:28:09.804849 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:09.804623 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-openshift-service-ca" pod "68ddb1ba-594c-4117-b6b2-2e7a06eb90de" (UID: "68ddb1ba-594c-4117-b6b2-2e7a06eb90de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:28:09.804994 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:09.804847 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:09.812082 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:09.812046 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-maas-api-tls" pod "68ddb1ba-594c-4117-b6b2-2e7a06eb90de" (UID: "68ddb1ba-594c-4117-b6b2-2e7a06eb90de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:28:09.812082 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:09.812066 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-metrics-certs" pod "68ddb1ba-594c-4117-b6b2-2e7a06eb90de" (UID: "68ddb1ba-594c-4117-b6b2-2e7a06eb90de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:28:09.812082 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:09.812079 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-kube-api-access-4mtbl" pod "68ddb1ba-594c-4117-b6b2-2e7a06eb90de" (UID: "68ddb1ba-594c-4117-b6b2-2e7a06eb90de"). InnerVolumeSpecName "kube-api-access-4mtbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:09.906349 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:09.906262 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:09.906562 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:09.906542 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:09.906721 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:09.906696 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mtbl\" (UniqueName: \"kubernetes.io/projected/68ddb1ba-594c-4117-b6b2-2e7a06eb90de-kube-api-access-4mtbl\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:10.038168 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:10.038135 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:10.220499 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:10.220466 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:10.381559 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:10.381517 4963 generic.go:356] "Generic (PLEG): container finished" podID="68ddb1ba-594c-4117-b6b2-2e7a06eb90de" containerID="b4f993b5b11c82514e098e00a0d0f0cfe9e7dbd6830f55c584b08ae85e2cbaa2" exitCode=0 Nov 03 21:28:10.382160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:10.381655 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" event={"ID":"68ddb1ba-594c-4117-b6b2-2e7a06eb90de","Type":"ContainerDied","Data":"b4f993b5b11c82514e098e00a0d0f0cfe9e7dbd6830f55c584b08ae85e2cbaa2"} Nov 03 21:28:10.382160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:10.381719 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" event={"ID":"68ddb1ba-594c-4117-b6b2-2e7a06eb90de","Type":"ContainerDied","Data":"e74273be336cd0f8cb937f5d346acb22ad32f4921fd741f9aafb573f5e049c4f"} Nov 03 21:28:10.382160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:10.381750 4963 scope.go:122] "RemoveContainer" containerID="b4f993b5b11c82514e098e00a0d0f0cfe9e7dbd6830f55c584b08ae85e2cbaa2" Nov 03 21:28:10.382160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:10.382035 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr" Nov 03 21:28:10.401996 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:10.401969 4963 scope.go:122] "RemoveContainer" containerID="b4f993b5b11c82514e098e00a0d0f0cfe9e7dbd6830f55c584b08ae85e2cbaa2" Nov 03 21:28:10.402343 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:28:10.402320 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4f993b5b11c82514e098e00a0d0f0cfe9e7dbd6830f55c584b08ae85e2cbaa2\": container with ID starting with b4f993b5b11c82514e098e00a0d0f0cfe9e7dbd6830f55c584b08ae85e2cbaa2 not found: ID does not exist" containerID="b4f993b5b11c82514e098e00a0d0f0cfe9e7dbd6830f55c584b08ae85e2cbaa2" Nov 03 21:28:10.402407 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:10.402352 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4f993b5b11c82514e098e00a0d0f0cfe9e7dbd6830f55c584b08ae85e2cbaa2"} err="failed to get container status \"b4f993b5b11c82514e098e00a0d0f0cfe9e7dbd6830f55c584b08ae85e2cbaa2\": rpc error: code = NotFound desc = could not find container \"b4f993b5b11c82514e098e00a0d0f0cfe9e7dbd6830f55c584b08ae85e2cbaa2\": container with ID starting with b4f993b5b11c82514e098e00a0d0f0cfe9e7dbd6830f55c584b08ae85e2cbaa2 not found: ID does not exist" Nov 03 21:28:10.548072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:10.547985 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr"] Nov 03 21:28:10.551855 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:10.551824 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eae138-86796b77b9-dfbpr"] Nov 03 21:28:10.968142 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:10.968107 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68ddb1ba-594c-4117-b6b2-2e7a06eb90de" path="/var/lib/kubelet/pods/68ddb1ba-594c-4117-b6b2-2e7a06eb90de/volumes" Nov 03 21:28:11.325941 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:11.325824 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:11.554053 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:11.554017 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:11.768760 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:11.768725 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:11.954708 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:11.954680 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:12.527959 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:28:12.527893 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:28:12.572593 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:12.572558 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:14.518345 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:14.518305 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:14.706511 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:14.706480 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:16.124639 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:16.124604 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:17.895577 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:17.895541 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:18.613179 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:18.613118 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:18.792523 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:18.792479 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:18.978959 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:18.978923 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:20.282481 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:20.282443 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:20.806984 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:20.806946 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:20.990559 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:20.990527 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:22.165418 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:22.165388 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:23.851405 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:23.851373 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:24.489623 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:24.489597 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:24.671259 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:24.671223 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:26.020469 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:26.020413 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:26.500703 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:26.500664 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:26.693422 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:26.693388 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:26.953383 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:26.953346 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:28.046934 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:28.046880 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:28.662740 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:28.662701 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:28.875791 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:28.875761 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:31.788418 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:31.788376 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:32.705780 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:32.705751 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:32.895270 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:32.895237 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:34.707577 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:34.707540 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:34.893980 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:34.893949 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:35.911662 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:35.911626 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:36.102742 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:36.102709 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:37.928534 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:37.928499 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:38.790782 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:38.790743 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:38.973552 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:38.973497 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:40.340537 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:40.340451 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:40.841089 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:40.841051 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:41.692092 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:41.692056 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:42.529604 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:28:42.529563 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:28:42.856804 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:42.856722 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:47.546462 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:47.546423 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:48.402539 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:48.402509 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:52.403550 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:52.403476 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:54.186381 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:54.186339 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:54.372808 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:54.372773 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:57.391963 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:28:57.391925 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:01.908541 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:01.908505 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:03.546058 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:03.546019 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:06.415009 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:06.414944 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:08.735062 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.735027 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp"] Nov 03 21:29:08.735540 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.735517 4963 cpu_manager.go:414] "containerMap: removing container" podUID="faeed472-64ed-40e2-aee1-0703462ccbe1" containerName="istio-proxy" Nov 03 21:29:08.735540 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.735534 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faeed472-64ed-40e2-aee1-0703462ccbe1" containerName="istio-proxy" Nov 03 21:29:08.735721 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.735553 4963 cpu_manager.go:414] "containerMap: removing container" podUID="295db5cb-7753-4ca5-aecb-cbb3a1851a87" containerName="main" Nov 03 21:29:08.735721 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.735561 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295db5cb-7753-4ca5-aecb-cbb3a1851a87" containerName="main" Nov 03 21:29:08.735721 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.735579 4963 cpu_manager.go:414] "containerMap: removing container" podUID="68ddb1ba-594c-4117-b6b2-2e7a06eb90de" containerName="maas-api" Nov 03 21:29:08.735721 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.735587 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ddb1ba-594c-4117-b6b2-2e7a06eb90de" containerName="maas-api" Nov 03 21:29:08.735721 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.735597 4963 cpu_manager.go:414] "containerMap: removing container" podUID="30c75fe2-5f87-4852-9f1f-703f4ff01304" containerName="istio-proxy" Nov 03 21:29:08.735721 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.735602 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c75fe2-5f87-4852-9f1f-703f4ff01304" containerName="istio-proxy" Nov 03 21:29:08.735721 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.735623 4963 cpu_manager.go:414] "containerMap: removing container" podUID="035dbca4-b720-4fc5-84a6-5b3e421c20d5" containerName="maas-api" Nov 03 21:29:08.735721 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.735628 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="035dbca4-b720-4fc5-84a6-5b3e421c20d5" containerName="maas-api" Nov 03 21:29:08.736047 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.735727 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="faeed472-64ed-40e2-aee1-0703462ccbe1" containerName="istio-proxy" Nov 03 21:29:08.736047 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.735740 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="295db5cb-7753-4ca5-aecb-cbb3a1851a87" containerName="main" Nov 03 21:29:08.736047 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.735747 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ddb1ba-594c-4117-b6b2-2e7a06eb90de" containerName="maas-api" Nov 03 21:29:08.736047 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.735753 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c75fe2-5f87-4852-9f1f-703f4ff01304" containerName="istio-proxy" Nov 03 21:29:08.736047 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.735761 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="035dbca4-b720-4fc5-84a6-5b3e421c20d5" containerName="maas-api" Nov 03 21:29:08.738876 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.738859 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.833441 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.833391 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.833441 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.833438 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.833816 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.833572 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.833816 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.833645 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0afb830-dc6c-40af-b463-3edeffe576e4-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.833816 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.833725 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.834023 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.833824 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.834023 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.833862 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.834023 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.833883 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flblh\" (UniqueName: \"kubernetes.io/projected/f0afb830-dc6c-40af-b463-3edeffe576e4-kube-api-access-flblh\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.834023 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.833970 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.929746 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.929707 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp"] Nov 03 21:29:08.935297 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.935257 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.935485 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.935325 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.935485 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.935389 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.935485 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.935454 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0afb830-dc6c-40af-b463-3edeffe576e4-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.935657 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.935495 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.935657 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.935578 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.935657 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.935609 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.935657 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.935639 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flblh\" (UniqueName: \"kubernetes.io/projected/f0afb830-dc6c-40af-b463-3edeffe576e4-kube-api-access-flblh\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.935850 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.935683 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.936142 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.936106 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.936238 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.936114 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.936238 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.936171 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.936713 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.936687 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0afb830-dc6c-40af-b463-3edeffe576e4-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.936823 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.936689 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.939522 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.939496 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.942202 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.942172 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.993415 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.993338 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:08.993960 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:08.993940 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flblh\" (UniqueName: \"kubernetes.io/projected/f0afb830-dc6c-40af-b463-3edeffe576e4-kube-api-access-flblh\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:09.054004 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:09.053965 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:09.486130 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:09.486095 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp"] Nov 03 21:29:09.491363 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:29:09.491335 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0afb830_dc6c_40af_b463_3edeffe576e4.slice/crio-b315a69ffc637253de667358bd79c29add8076cae80ae73eebff69ffd18b6c70 WatchSource:0}: Error finding container b315a69ffc637253de667358bd79c29add8076cae80ae73eebff69ffd18b6c70: Status 404 returned error can't find the container with id b315a69ffc637253de667358bd79c29add8076cae80ae73eebff69ffd18b6c70 Nov 03 21:29:09.493709 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:09.493674 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:29:09.493820 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:09.493774 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:29:09.493887 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:09.493816 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:29:09.754108 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:09.754020 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" event={"ID":"f0afb830-dc6c-40af-b463-3edeffe576e4","Type":"ContainerStarted","Data":"3ea222969033222b330e02b52f4f049a60c51fe88b6ea0f21daf424dc420d1ca"} Nov 03 21:29:09.754108 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:09.754103 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" event={"ID":"f0afb830-dc6c-40af-b463-3edeffe576e4","Type":"ContainerStarted","Data":"b315a69ffc637253de667358bd79c29add8076cae80ae73eebff69ffd18b6c70"} Nov 03 21:29:09.871727 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:09.871663 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" podStartSLOduration=1.8716423199999999 podStartE2EDuration="1.87164232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:09.869072636 +0000 UTC m=+2499.029144909" watchObservedRunningTime="2026-11-03 21:29:09.87164232 +0000 UTC m=+2499.031714593" Nov 03 21:29:10.054590 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:10.054512 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:10.056265 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:10.056238 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 21:29:10.056406 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:10.056293 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" podUID="f0afb830-dc6c-40af-b463-3edeffe576e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 21:29:11.033074 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:11.033039 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:11.055518 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:11.055462 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 21:29:11.055848 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:11.055796 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" podUID="f0afb830-dc6c-40af-b463-3edeffe576e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 21:29:11.709265 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:11.709226 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl"] Nov 03 21:29:11.709491 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:11.709429 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" podUID="955fcff3-0d65-4115-916a-18c1506a3aba" containerName="istio-proxy" containerID="cri-o://6c53948cee8c0689fdbbde22ff64af1a887ef0d2606b959f8bb1fd86446339e9" gracePeriod=30 Nov 03 21:29:12.054512 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:12.054415 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 21:29:12.054512 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:12.054489 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" podUID="f0afb830-dc6c-40af-b463-3edeffe576e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 21:29:12.532656 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:29:12.532618 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:29:13.055331 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:13.055287 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 21:29:13.055797 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:13.055362 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" podUID="f0afb830-dc6c-40af-b463-3edeffe576e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 21:29:14.054796 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:14.054756 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 21:29:14.055125 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:14.055094 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" podUID="f0afb830-dc6c-40af-b463-3edeffe576e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 21:29:14.569372 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:14.569336 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:29:14.569841 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:14.569396 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" podUID="955fcff3-0d65-4115-916a-18c1506a3aba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:29:15.054317 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:15.054280 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 21:29:15.054500 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:15.054338 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" podUID="f0afb830-dc6c-40af-b463-3edeffe576e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 21:29:15.627745 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:15.627705 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:15.811640 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:15.811604 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:16.004433 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:16.004402 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:16.054951 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:16.054890 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 21:29:16.055120 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:16.054970 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" podUID="f0afb830-dc6c-40af-b463-3edeffe576e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 21:29:16.846922 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:16.846149 4963 generic.go:356] "Generic (PLEG): container finished" podID="955fcff3-0d65-4115-916a-18c1506a3aba" containerID="6c53948cee8c0689fdbbde22ff64af1a887ef0d2606b959f8bb1fd86446339e9" exitCode=0 Nov 03 21:29:16.846922 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:16.846194 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" event={"ID":"955fcff3-0d65-4115-916a-18c1506a3aba","Type":"ContainerDied","Data":"6c53948cee8c0689fdbbde22ff64af1a887ef0d2606b959f8bb1fd86446339e9"} Nov 03 21:29:17.046493 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.046463 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:29:17.181567 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.181530 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-istio-data\") pod \"955fcff3-0d65-4115-916a-18c1506a3aba\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " Nov 03 21:29:17.181785 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.181578 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-workload-socket\") pod \"955fcff3-0d65-4115-916a-18c1506a3aba\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " Nov 03 21:29:17.181785 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.181618 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-workload-certs\") pod \"955fcff3-0d65-4115-916a-18c1506a3aba\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " Nov 03 21:29:17.181785 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.181665 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-credential-socket\") pod \"955fcff3-0d65-4115-916a-18c1506a3aba\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " Nov 03 21:29:17.181785 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.181706 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/955fcff3-0d65-4115-916a-18c1506a3aba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955fcff3-0d65-4115-916a-18c1506a3aba-istiod-ca-cert\") pod \"955fcff3-0d65-4115-916a-18c1506a3aba\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " Nov 03 21:29:17.181785 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.181765 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-istio-envoy\") pod \"955fcff3-0d65-4115-916a-18c1506a3aba\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " Nov 03 21:29:17.182064 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.181870 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955fcff3-0d65-4115-916a-18c1506a3aba-kube-api-access-ccxrq\" (UniqueName: \"kubernetes.io/projected/955fcff3-0d65-4115-916a-18c1506a3aba-kube-api-access-ccxrq\") pod \"955fcff3-0d65-4115-916a-18c1506a3aba\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " Nov 03 21:29:17.182064 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.181917 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/955fcff3-0d65-4115-916a-18c1506a3aba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955fcff3-0d65-4115-916a-18c1506a3aba-istio-podinfo\") pod \"955fcff3-0d65-4115-916a-18c1506a3aba\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " Nov 03 21:29:17.182064 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.182009 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955fcff3-0d65-4115-916a-18c1506a3aba-istio-token\" (UniqueName: \"kubernetes.io/projected/955fcff3-0d65-4115-916a-18c1506a3aba-istio-token\") pod \"955fcff3-0d65-4115-916a-18c1506a3aba\" (UID: \"955fcff3-0d65-4115-916a-18c1506a3aba\") " Nov 03 21:29:17.182064 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.181866 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-istio-data" pod "955fcff3-0d65-4115-916a-18c1506a3aba" (UID: "955fcff3-0d65-4115-916a-18c1506a3aba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:29:17.182064 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.182026 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-workload-socket" pod "955fcff3-0d65-4115-916a-18c1506a3aba" (UID: "955fcff3-0d65-4115-916a-18c1506a3aba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:29:17.182342 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.182157 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-credential-socket" pod "955fcff3-0d65-4115-916a-18c1506a3aba" (UID: "955fcff3-0d65-4115-916a-18c1506a3aba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:29:17.182342 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.182239 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/955fcff3-0d65-4115-916a-18c1506a3aba-istiod-ca-cert" pod "955fcff3-0d65-4115-916a-18c1506a3aba" (UID: "955fcff3-0d65-4115-916a-18c1506a3aba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:29:17.182449 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.182439 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:17.182504 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.182453 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:17.182504 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.182468 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:17.182504 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.182477 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955fcff3-0d65-4115-916a-18c1506a3aba-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:17.182681 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.182635 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-workload-certs" pod "955fcff3-0d65-4115-916a-18c1506a3aba" (UID: "955fcff3-0d65-4115-916a-18c1506a3aba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:29:17.184961 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.184935 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-istio-envoy" pod "955fcff3-0d65-4115-916a-18c1506a3aba" (UID: "955fcff3-0d65-4115-916a-18c1506a3aba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:29:17.185083 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.184985 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/955fcff3-0d65-4115-916a-18c1506a3aba-istio-podinfo" pod "955fcff3-0d65-4115-916a-18c1506a3aba" (UID: "955fcff3-0d65-4115-916a-18c1506a3aba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:29:17.185143 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.185128 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955fcff3-0d65-4115-916a-18c1506a3aba-kube-api-access-ccxrq" pod "955fcff3-0d65-4115-916a-18c1506a3aba" (UID: "955fcff3-0d65-4115-916a-18c1506a3aba"). InnerVolumeSpecName "kube-api-access-ccxrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:29:17.185264 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.185242 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955fcff3-0d65-4115-916a-18c1506a3aba-istio-token" pod "955fcff3-0d65-4115-916a-18c1506a3aba" (UID: "955fcff3-0d65-4115-916a-18c1506a3aba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:29:17.283992 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.283874 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccxrq\" (UniqueName: \"kubernetes.io/projected/955fcff3-0d65-4115-916a-18c1506a3aba-kube-api-access-ccxrq\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:17.283992 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.283922 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955fcff3-0d65-4115-916a-18c1506a3aba-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:17.283992 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.283933 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/955fcff3-0d65-4115-916a-18c1506a3aba-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:17.283992 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.283944 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:17.283992 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.283953 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955fcff3-0d65-4115-916a-18c1506a3aba-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:17.854329 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.854298 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" Nov 03 21:29:17.854765 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.854329 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl" event={"ID":"955fcff3-0d65-4115-916a-18c1506a3aba","Type":"ContainerDied","Data":"cb6f2832a47aa395ebb36e7a48a0a4a462631dac5073be7fa2fd93c6816164fc"} Nov 03 21:29:17.854765 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:17.854390 4963 scope.go:122] "RemoveContainer" containerID="6c53948cee8c0689fdbbde22ff64af1a887ef0d2606b959f8bb1fd86446339e9" Nov 03 21:29:18.060082 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:18.060048 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl"] Nov 03 21:29:18.096064 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:18.096018 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:29:18.096255 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:18.096093 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" podUID="f0afb830-dc6c-40af-b463-3edeffe576e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:29:18.108190 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:18.108083 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-dxqwl"] Nov 03 21:29:18.364146 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:18.364023 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:18.364529 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:18.364502 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:18.365693 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:18.365670 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:29:18.967835 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:18.967794 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="955fcff3-0d65-4115-916a-18c1506a3aba" path="/var/lib/kubelet/pods/955fcff3-0d65-4115-916a-18c1506a3aba/volumes" Nov 03 21:29:20.357330 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:20.357294 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:24.890126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:24.890085 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:25.227571 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:25.227533 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6"] Nov 03 21:29:25.227785 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:25.227759 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" podUID="6ed228cb-abae-423f-b7f2-3dd6f8a1096f" containerName="istio-proxy" containerID="cri-o://c122f8eb142ff2a52e9eb2e6908940a578504cdb3a9ba3178b21a30a0b6e0deb" gracePeriod=30 Nov 03 21:29:26.945987 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:26.945950 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:29.417544 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:29.417506 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:30.501862 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.501796 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:29:30.541780 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.541745 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-workload-certs\") pod \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " Nov 03 21:29:30.541780 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.541782 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-data\") pod \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " Nov 03 21:29:30.542126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.541823 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-kube-api-access-fdhwr\" (UniqueName: \"kubernetes.io/projected/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-kube-api-access-fdhwr\") pod \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " Nov 03 21:29:30.542126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.541855 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-envoy\") pod \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " Nov 03 21:29:30.542126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.541879 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-workload-socket\") pod \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " Nov 03 21:29:30.542126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.541917 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-credential-socket\") pod \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " Nov 03 21:29:30.542126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.542076 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-workload-certs" pod "6ed228cb-abae-423f-b7f2-3dd6f8a1096f" (UID: "6ed228cb-abae-423f-b7f2-3dd6f8a1096f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:29:30.542378 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.542165 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-data" pod "6ed228cb-abae-423f-b7f2-3dd6f8a1096f" (UID: "6ed228cb-abae-423f-b7f2-3dd6f8a1096f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:29:30.542378 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.542266 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-workload-socket" pod "6ed228cb-abae-423f-b7f2-3dd6f8a1096f" (UID: "6ed228cb-abae-423f-b7f2-3dd6f8a1096f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:29:30.542480 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.542375 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-credential-socket" pod "6ed228cb-abae-423f-b7f2-3dd6f8a1096f" (UID: "6ed228cb-abae-423f-b7f2-3dd6f8a1096f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:29:30.542480 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.542451 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:30.542480 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.542464 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:30.542480 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.542474 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:30.542636 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.542482 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:30.544228 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.544192 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-envoy" pod "6ed228cb-abae-423f-b7f2-3dd6f8a1096f" (UID: "6ed228cb-abae-423f-b7f2-3dd6f8a1096f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:29:30.544439 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.544420 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-kube-api-access-fdhwr" pod "6ed228cb-abae-423f-b7f2-3dd6f8a1096f" (UID: "6ed228cb-abae-423f-b7f2-3dd6f8a1096f"). InnerVolumeSpecName "kube-api-access-fdhwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:29:30.643267 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.643225 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-token\" (UniqueName: \"kubernetes.io/projected/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-token\") pod \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " Nov 03 21:29:30.643470 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.643301 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-podinfo\") pod \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " Nov 03 21:29:30.643536 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.643518 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istiod-ca-cert\") pod \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\" (UID: \"6ed228cb-abae-423f-b7f2-3dd6f8a1096f\") " Nov 03 21:29:30.643869 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.643846 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdhwr\" (UniqueName: \"kubernetes.io/projected/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-kube-api-access-fdhwr\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:30.643869 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.643853 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istiod-ca-cert" pod "6ed228cb-abae-423f-b7f2-3dd6f8a1096f" (UID: "6ed228cb-abae-423f-b7f2-3dd6f8a1096f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:29:30.643869 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.643870 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:30.646105 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.646073 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-podinfo" pod "6ed228cb-abae-423f-b7f2-3dd6f8a1096f" (UID: "6ed228cb-abae-423f-b7f2-3dd6f8a1096f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:29:30.646195 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.646101 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-token" pod "6ed228cb-abae-423f-b7f2-3dd6f8a1096f" (UID: "6ed228cb-abae-423f-b7f2-3dd6f8a1096f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:29:30.745394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.745348 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:30.745394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.745389 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:30.745632 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.745432 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed228cb-abae-423f-b7f2-3dd6f8a1096f-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:30.942043 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.942013 4963 generic.go:356] "Generic (PLEG): container finished" podID="6ed228cb-abae-423f-b7f2-3dd6f8a1096f" containerID="c122f8eb142ff2a52e9eb2e6908940a578504cdb3a9ba3178b21a30a0b6e0deb" exitCode=0 Nov 03 21:29:30.942236 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.942060 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" Nov 03 21:29:30.942236 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.942091 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" event={"ID":"6ed228cb-abae-423f-b7f2-3dd6f8a1096f","Type":"ContainerDied","Data":"c122f8eb142ff2a52e9eb2e6908940a578504cdb3a9ba3178b21a30a0b6e0deb"} Nov 03 21:29:30.942236 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.942171 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6" event={"ID":"6ed228cb-abae-423f-b7f2-3dd6f8a1096f","Type":"ContainerDied","Data":"342270b7857e62b676c42ae581a309923ecb1af4d8d17d7ed073898d7b378688"} Nov 03 21:29:30.942236 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.942195 4963 scope.go:122] "RemoveContainer" containerID="c122f8eb142ff2a52e9eb2e6908940a578504cdb3a9ba3178b21a30a0b6e0deb" Nov 03 21:29:30.954491 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.954462 4963 scope.go:122] "RemoveContainer" containerID="c122f8eb142ff2a52e9eb2e6908940a578504cdb3a9ba3178b21a30a0b6e0deb" Nov 03 21:29:30.954837 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:29:30.954816 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c122f8eb142ff2a52e9eb2e6908940a578504cdb3a9ba3178b21a30a0b6e0deb\": container with ID starting with c122f8eb142ff2a52e9eb2e6908940a578504cdb3a9ba3178b21a30a0b6e0deb not found: ID does not exist" containerID="c122f8eb142ff2a52e9eb2e6908940a578504cdb3a9ba3178b21a30a0b6e0deb" Nov 03 21:29:30.954917 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:30.954850 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c122f8eb142ff2a52e9eb2e6908940a578504cdb3a9ba3178b21a30a0b6e0deb"} err="failed to get container status \"c122f8eb142ff2a52e9eb2e6908940a578504cdb3a9ba3178b21a30a0b6e0deb\": rpc error: code = NotFound desc = could not find container \"c122f8eb142ff2a52e9eb2e6908940a578504cdb3a9ba3178b21a30a0b6e0deb\": container with ID starting with c122f8eb142ff2a52e9eb2e6908940a578504cdb3a9ba3178b21a30a0b6e0deb not found: ID does not exist" Nov 03 21:29:31.103662 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:31.103627 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6"] Nov 03 21:29:31.151148 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:31.151114 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-67kl6"] Nov 03 21:29:32.967195 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:32.967150 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ed228cb-abae-423f-b7f2-3dd6f8a1096f" path="/var/lib/kubelet/pods/6ed228cb-abae-423f-b7f2-3dd6f8a1096f/volumes" Nov 03 21:29:33.349979 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:33.349860 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:34.439208 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:34.439167 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:35.019026 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:35.018990 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:35.206388 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:35.206349 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:35.392089 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:35.391989 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:37.013581 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:37.013547 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:37.266408 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:37.266332 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:38.533078 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:38.533040 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:39.059479 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:39.059441 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:39.972031 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:39.971899 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:40.157423 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:40.157392 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:40.371596 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:40.371519 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:40.563827 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:40.563789 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:41.428940 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:41.425937 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:41.681764 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:41.681679 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:42.341014 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:42.340973 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:42.527580 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:29:42.527540 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:29:43.033232 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:43.033199 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:44.931583 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:44.931542 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:45.118556 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:45.118520 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:46.541914 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:46.541881 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:46.744780 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:46.744751 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:48.064761 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:48.064727 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:48.665221 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:48.665180 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:49.995128 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:49.995085 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:54.606782 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:54.606716 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:57.143211 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.143171 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:57.326285 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.326250 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:57.451176 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.451145 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx"] Nov 03 21:29:57.451729 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.451714 4963 cpu_manager.go:414] "containerMap: removing container" podUID="955fcff3-0d65-4115-916a-18c1506a3aba" containerName="istio-proxy" Nov 03 21:29:57.451836 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.451729 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955fcff3-0d65-4115-916a-18c1506a3aba" containerName="istio-proxy" Nov 03 21:29:57.451836 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.451765 4963 cpu_manager.go:414] "containerMap: removing container" podUID="6ed228cb-abae-423f-b7f2-3dd6f8a1096f" containerName="istio-proxy" Nov 03 21:29:57.451836 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.451771 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed228cb-abae-423f-b7f2-3dd6f8a1096f" containerName="istio-proxy" Nov 03 21:29:57.452065 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.451852 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed228cb-abae-423f-b7f2-3dd6f8a1096f" containerName="istio-proxy" Nov 03 21:29:57.452065 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.451866 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="955fcff3-0d65-4115-916a-18c1506a3aba" containerName="istio-proxy" Nov 03 21:29:57.455447 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.455426 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.589248 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.589212 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.589447 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.589351 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q66s\" (UniqueName: \"kubernetes.io/projected/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-kube-api-access-2q66s\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.589447 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.589384 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.589447 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.589413 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.589627 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.589563 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.589627 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.589620 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.589696 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.589671 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.589731 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.589705 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.589765 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.589742 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.661634 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.661595 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx"] Nov 03 21:29:57.691342 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.691302 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.691564 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.691373 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.691564 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.691429 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.691564 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.691467 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.691564 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.691502 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.691811 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.691570 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.691811 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.691708 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q66s\" (UniqueName: \"kubernetes.io/projected/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-kube-api-access-2q66s\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.691811 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.691742 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.691811 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.691774 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.694770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.693149 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.694770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.693345 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.694770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.694651 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.695130 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.695105 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.695703 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.695678 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.697450 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.697426 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.698267 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.698246 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.753058 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.752984 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.753231 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.753130 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q66s\" (UniqueName: \"kubernetes.io/projected/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-kube-api-access-2q66s\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:57.770080 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:57.770036 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:58.203971 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:58.203939 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx"] Nov 03 21:29:58.210404 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:29:58.210372 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b117e23_def9_49c7_a6bd_db4ccb5c1a12.slice/crio-52d63c6954b70563647a66891b693d742d9491eb0e2be96e6659bfdb4b1ec645 WatchSource:0}: Error finding container 52d63c6954b70563647a66891b693d742d9491eb0e2be96e6659bfdb4b1ec645: Status 404 returned error can't find the container with id 52d63c6954b70563647a66891b693d742d9491eb0e2be96e6659bfdb4b1ec645 Nov 03 21:29:58.213202 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:58.213167 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:29:58.213304 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:58.213240 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:29:58.213304 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:58.213272 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:29:59.122509 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:59.122472 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:59.235941 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:59.235854 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" event={"ID":"7b117e23-def9-49c7-a6bd-db4ccb5c1a12","Type":"ContainerStarted","Data":"a1c35caeb52f410821f40603934300bc2258c6e5e2bd7089952d88dc52b930fd"} Nov 03 21:29:59.236611 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:59.236580 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" event={"ID":"7b117e23-def9-49c7-a6bd-db4ccb5c1a12","Type":"ContainerStarted","Data":"52d63c6954b70563647a66891b693d742d9491eb0e2be96e6659bfdb4b1ec645"} Nov 03 21:29:59.412153 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:59.411998 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" podStartSLOduration=2.411958102 podStartE2EDuration="2.411958102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:59.410003038 +0000 UTC m=+2548.570075307" watchObservedRunningTime="2026-11-03 21:29:59.411958102 +0000 UTC m=+2548.572030375" Nov 03 21:29:59.771163 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:59.771133 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:29:59.772955 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:59.772927 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:29:59.773107 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:29:59.772980 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" podUID="7b117e23-def9-49c7-a6bd-db4ccb5c1a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 21:30:00.164217 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.164042 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg"] Nov 03 21:30:00.168236 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.168212 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" Nov 03 21:30:00.202184 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.201962 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk"] Nov 03 21:30:00.218926 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.216187 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" Nov 03 21:30:00.222938 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.219881 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr"] Nov 03 21:30:00.224732 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.224680 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" Nov 03 21:30:00.229686 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.229659 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\"" Nov 03 21:30:00.252521 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.252488 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/726fefeb-77f7-49bd-b9b0-c4c4425ab096-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk\" (UID: \"726fefeb-77f7-49bd-b9b0-c4c4425ab096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" Nov 03 21:30:00.254837 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.254809 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5xkt\" (UniqueName: \"kubernetes.io/projected/169f11b5-0e68-4242-8fee-82d85da91b3b-kube-api-access-l5xkt\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr\" (UID: \"169f11b5-0e68-4242-8fee-82d85da91b3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" Nov 03 21:30:00.255106 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.255085 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66smt\" (UniqueName: \"kubernetes.io/projected/726fefeb-77f7-49bd-b9b0-c4c4425ab096-kube-api-access-66smt\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk\" (UID: \"726fefeb-77f7-49bd-b9b0-c4c4425ab096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" Nov 03 21:30:00.255249 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.255235 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93239ed2-3967-4b97-b37e-6993cad96297-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg\" (UID: \"93239ed2-3967-4b97-b37e-6993cad96297\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" Nov 03 21:30:00.255394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.255380 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nqmv\" (UniqueName: \"kubernetes.io/projected/93239ed2-3967-4b97-b37e-6993cad96297-kube-api-access-5nqmv\") pod \"maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg\" (UID: \"93239ed2-3967-4b97-b37e-6993cad96297\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" Nov 03 21:30:00.256023 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.256004 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169f11b5-0e68-4242-8fee-82d85da91b3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr\" (UID: \"169f11b5-0e68-4242-8fee-82d85da91b3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" Nov 03 21:30:00.266846 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.266764 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg"] Nov 03 21:30:00.270839 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.270781 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk"] Nov 03 21:30:00.274530 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.274507 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr"] Nov 03 21:30:00.357507 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.357461 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/726fefeb-77f7-49bd-b9b0-c4c4425ab096-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk\" (UID: \"726fefeb-77f7-49bd-b9b0-c4c4425ab096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" Nov 03 21:30:00.357719 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.357561 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5xkt\" (UniqueName: \"kubernetes.io/projected/169f11b5-0e68-4242-8fee-82d85da91b3b-kube-api-access-l5xkt\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr\" (UID: \"169f11b5-0e68-4242-8fee-82d85da91b3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" Nov 03 21:30:00.357719 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.357592 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66smt\" (UniqueName: \"kubernetes.io/projected/726fefeb-77f7-49bd-b9b0-c4c4425ab096-kube-api-access-66smt\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk\" (UID: \"726fefeb-77f7-49bd-b9b0-c4c4425ab096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" Nov 03 21:30:00.357719 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.357646 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93239ed2-3967-4b97-b37e-6993cad96297-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg\" (UID: \"93239ed2-3967-4b97-b37e-6993cad96297\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" Nov 03 21:30:00.357719 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.357713 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nqmv\" (UniqueName: \"kubernetes.io/projected/93239ed2-3967-4b97-b37e-6993cad96297-kube-api-access-5nqmv\") pod \"maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg\" (UID: \"93239ed2-3967-4b97-b37e-6993cad96297\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" Nov 03 21:30:00.358006 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.357745 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169f11b5-0e68-4242-8fee-82d85da91b3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr\" (UID: \"169f11b5-0e68-4242-8fee-82d85da91b3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" Nov 03 21:30:00.358215 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.358195 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/726fefeb-77f7-49bd-b9b0-c4c4425ab096-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk\" (UID: \"726fefeb-77f7-49bd-b9b0-c4c4425ab096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" Nov 03 21:30:00.358515 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.358492 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169f11b5-0e68-4242-8fee-82d85da91b3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr\" (UID: \"169f11b5-0e68-4242-8fee-82d85da91b3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" Nov 03 21:30:00.358599 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.358556 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93239ed2-3967-4b97-b37e-6993cad96297-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg\" (UID: \"93239ed2-3967-4b97-b37e-6993cad96297\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" Nov 03 21:30:00.425248 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.425161 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nqmv\" (UniqueName: \"kubernetes.io/projected/93239ed2-3967-4b97-b37e-6993cad96297-kube-api-access-5nqmv\") pod \"maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg\" (UID: \"93239ed2-3967-4b97-b37e-6993cad96297\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" Nov 03 21:30:00.425850 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.425819 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5xkt\" (UniqueName: \"kubernetes.io/projected/169f11b5-0e68-4242-8fee-82d85da91b3b-kube-api-access-l5xkt\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr\" (UID: \"169f11b5-0e68-4242-8fee-82d85da91b3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" Nov 03 21:30:00.426015 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.425870 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66smt\" (UniqueName: \"kubernetes.io/projected/726fefeb-77f7-49bd-b9b0-c4c4425ab096-kube-api-access-66smt\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk\" (UID: \"726fefeb-77f7-49bd-b9b0-c4c4425ab096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" Nov 03 21:30:00.496324 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.495085 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" Nov 03 21:30:00.554789 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.554726 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" Nov 03 21:30:00.567294 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.567079 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" Nov 03 21:30:00.774080 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.773157 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:30:00.774080 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.773230 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" podUID="7b117e23-def9-49c7-a6bd-db4ccb5c1a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 21:30:00.935825 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:00.935394 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg"] Nov 03 21:30:00.945426 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:30:00.944031 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93239ed2_3967_4b97_b37e_6993cad96297.slice/crio-6c3f85223f9d04247615b70a8d26abe5bd9fcea689cce19c9300afc24fd0830d WatchSource:0}: Error finding container 6c3f85223f9d04247615b70a8d26abe5bd9fcea689cce19c9300afc24fd0830d: Status 404 returned error can't find the container with id 6c3f85223f9d04247615b70a8d26abe5bd9fcea689cce19c9300afc24fd0830d Nov 03 21:30:01.096476 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:30:01.096427 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726fefeb_77f7_49bd_b9b0_c4c4425ab096.slice/crio-82d33a3a23acd9037af5ccb2ad74c42006823df59fec434b69b2e9e7ed424da8 WatchSource:0}: Error finding container 82d33a3a23acd9037af5ccb2ad74c42006823df59fec434b69b2e9e7ed424da8: Status 404 returned error can't find the container with id 82d33a3a23acd9037af5ccb2ad74c42006823df59fec434b69b2e9e7ed424da8 Nov 03 21:30:01.099925 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:01.098857 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk"] Nov 03 21:30:01.133619 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:01.133563 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr"] Nov 03 21:30:01.150362 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:30:01.150328 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169f11b5_0e68_4242_8fee_82d85da91b3b.slice/crio-7e3bf8dec7af75df806faa6544f69e9c4bc0f2e3f422e0154d626d2fc82c27cf WatchSource:0}: Error finding container 7e3bf8dec7af75df806faa6544f69e9c4bc0f2e3f422e0154d626d2fc82c27cf: Status 404 returned error can't find the container with id 7e3bf8dec7af75df806faa6544f69e9c4bc0f2e3f422e0154d626d2fc82c27cf Nov 03 21:30:01.270733 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:01.270687 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" event={"ID":"726fefeb-77f7-49bd-b9b0-c4c4425ab096","Type":"ContainerStarted","Data":"82d33a3a23acd9037af5ccb2ad74c42006823df59fec434b69b2e9e7ed424da8"} Nov 03 21:30:01.272089 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:01.272057 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" event={"ID":"169f11b5-0e68-4242-8fee-82d85da91b3b","Type":"ContainerStarted","Data":"7e3bf8dec7af75df806faa6544f69e9c4bc0f2e3f422e0154d626d2fc82c27cf"} Nov 03 21:30:01.273613 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:01.273581 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" event={"ID":"93239ed2-3967-4b97-b37e-6993cad96297","Type":"ContainerStarted","Data":"479063ce7d6f6f8e2244e9fb6c4f7fa3ad9eee1067c255985c542ca5327a13c1"} Nov 03 21:30:01.273722 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:01.273634 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" event={"ID":"93239ed2-3967-4b97-b37e-6993cad96297","Type":"ContainerStarted","Data":"6c3f85223f9d04247615b70a8d26abe5bd9fcea689cce19c9300afc24fd0830d"} Nov 03 21:30:01.770509 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:01.770472 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:30:01.770697 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:01.770522 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" podUID="7b117e23-def9-49c7-a6bd-db4ccb5c1a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 21:30:01.911596 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:01.911564 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:02.286929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:02.285686 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" event={"ID":"726fefeb-77f7-49bd-b9b0-c4c4425ab096","Type":"ContainerStarted","Data":"2040ddfe522bcc13bab4f250931c8d0f4d886d989e04cf54c7f40aa54bb9b1a1"} Nov 03 21:30:02.318352 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:02.317604 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" event={"ID":"169f11b5-0e68-4242-8fee-82d85da91b3b","Type":"ContainerStarted","Data":"50875337ce952800b81e632a19c794fbd20012f1a9dd9e4252449d543918ae16"} Nov 03 21:30:02.321027 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:02.320990 4963 generic.go:356] "Generic (PLEG): container finished" podID="93239ed2-3967-4b97-b37e-6993cad96297" containerID="479063ce7d6f6f8e2244e9fb6c4f7fa3ad9eee1067c255985c542ca5327a13c1" exitCode=60 Nov 03 21:30:02.321189 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:02.321149 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" event={"ID":"93239ed2-3967-4b97-b37e-6993cad96297","Type":"ContainerDied","Data":"479063ce7d6f6f8e2244e9fb6c4f7fa3ad9eee1067c255985c542ca5327a13c1"} Nov 03 21:30:02.321568 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:02.321548 4963 scope.go:122] "RemoveContainer" containerID="479063ce7d6f6f8e2244e9fb6c4f7fa3ad9eee1067c255985c542ca5327a13c1" Nov 03 21:30:02.404935 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:02.400331 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" podStartSLOduration=2.400304225 podStartE2EDuration="2.400304225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:01.408243571 +0000 UTC m=+2550.568315844" watchObservedRunningTime="2026-11-03 21:30:02.400304225 +0000 UTC m=+2551.560376498" Nov 03 21:30:02.532653 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:02.532581 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" podStartSLOduration=2.532558512 podStartE2EDuration="2.532558512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:02.415376707 +0000 UTC m=+2551.575448979" watchObservedRunningTime="2026-11-03 21:30:02.532558512 +0000 UTC m=+2551.692630785" Nov 03 21:30:02.608984 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:02.608228 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:02.676795 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:02.676728 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" podStartSLOduration=2.676705502 podStartE2EDuration="2.676705502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:02.655164661 +0000 UTC m=+2551.815236934" watchObservedRunningTime="2026-11-03 21:30:02.676705502 +0000 UTC m=+2551.836777780" Nov 03 21:30:02.770541 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:02.770500 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:30:02.770704 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:02.770567 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" podUID="7b117e23-def9-49c7-a6bd-db4ccb5c1a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 21:30:02.926626 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:02.926590 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:03.329449 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:03.329409 4963 generic.go:356] "Generic (PLEG): container finished" podID="726fefeb-77f7-49bd-b9b0-c4c4425ab096" containerID="2040ddfe522bcc13bab4f250931c8d0f4d886d989e04cf54c7f40aa54bb9b1a1" exitCode=60 Nov 03 21:30:03.329851 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:03.329489 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" event={"ID":"726fefeb-77f7-49bd-b9b0-c4c4425ab096","Type":"ContainerDied","Data":"2040ddfe522bcc13bab4f250931c8d0f4d886d989e04cf54c7f40aa54bb9b1a1"} Nov 03 21:30:03.329851 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:03.329842 4963 scope.go:122] "RemoveContainer" containerID="2040ddfe522bcc13bab4f250931c8d0f4d886d989e04cf54c7f40aa54bb9b1a1" Nov 03 21:30:03.331204 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:03.331178 4963 generic.go:356] "Generic (PLEG): container finished" podID="169f11b5-0e68-4242-8fee-82d85da91b3b" containerID="50875337ce952800b81e632a19c794fbd20012f1a9dd9e4252449d543918ae16" exitCode=60 Nov 03 21:30:03.331322 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:03.331210 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" event={"ID":"169f11b5-0e68-4242-8fee-82d85da91b3b","Type":"ContainerDied","Data":"50875337ce952800b81e632a19c794fbd20012f1a9dd9e4252449d543918ae16"} Nov 03 21:30:03.331531 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:03.331515 4963 scope.go:122] "RemoveContainer" containerID="50875337ce952800b81e632a19c794fbd20012f1a9dd9e4252449d543918ae16" Nov 03 21:30:03.333449 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:03.333427 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" event={"ID":"93239ed2-3967-4b97-b37e-6993cad96297","Type":"ContainerStarted","Data":"1e677fd7f2f2dea3c07eada0ff272769a5ca567ef83cdf51e8d36f2773a494de"} Nov 03 21:30:03.771463 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:03.771426 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:30:03.771675 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:03.771486 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" podUID="7b117e23-def9-49c7-a6bd-db4ccb5c1a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 21:30:03.931700 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:03.931672 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:04.354631 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:04.354583 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" event={"ID":"726fefeb-77f7-49bd-b9b0-c4c4425ab096","Type":"ContainerStarted","Data":"4d956d1d30a528782e9c4699d179b08f847aa7dfa40ef1bde04300cdbbed49e3"} Nov 03 21:30:04.356885 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:04.356847 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" event={"ID":"169f11b5-0e68-4242-8fee-82d85da91b3b","Type":"ContainerStarted","Data":"a09f37edc46df77d7ff26cf0c24fc894dc62c3acb59f17c1c3412f342d3bd2e8"} Nov 03 21:30:04.359017 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:04.358989 4963 generic.go:356] "Generic (PLEG): container finished" podID="93239ed2-3967-4b97-b37e-6993cad96297" containerID="1e677fd7f2f2dea3c07eada0ff272769a5ca567ef83cdf51e8d36f2773a494de" exitCode=60 Nov 03 21:30:04.359183 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:04.359076 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" event={"ID":"93239ed2-3967-4b97-b37e-6993cad96297","Type":"ContainerDied","Data":"1e677fd7f2f2dea3c07eada0ff272769a5ca567ef83cdf51e8d36f2773a494de"} Nov 03 21:30:04.359183 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:04.359152 4963 scope.go:122] "RemoveContainer" containerID="479063ce7d6f6f8e2244e9fb6c4f7fa3ad9eee1067c255985c542ca5327a13c1" Nov 03 21:30:04.359360 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:04.359343 4963 scope.go:122] "RemoveContainer" containerID="1e677fd7f2f2dea3c07eada0ff272769a5ca567ef83cdf51e8d36f2773a494de" Nov 03 21:30:04.359657 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:30:04.359633 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg_odh-ai-gateway-infra(93239ed2-3967-4b97-b37e-6993cad96297)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" podUID="93239ed2-3967-4b97-b37e-6993cad96297" Nov 03 21:30:04.771505 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:04.771458 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 21:30:04.771697 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:04.771552 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" podUID="7b117e23-def9-49c7-a6bd-db4ccb5c1a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 21:30:05.367564 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:05.367520 4963 generic.go:356] "Generic (PLEG): container finished" podID="726fefeb-77f7-49bd-b9b0-c4c4425ab096" containerID="4d956d1d30a528782e9c4699d179b08f847aa7dfa40ef1bde04300cdbbed49e3" exitCode=60 Nov 03 21:30:05.368051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:05.367632 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" event={"ID":"726fefeb-77f7-49bd-b9b0-c4c4425ab096","Type":"ContainerDied","Data":"4d956d1d30a528782e9c4699d179b08f847aa7dfa40ef1bde04300cdbbed49e3"} Nov 03 21:30:05.368051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:05.367682 4963 scope.go:122] "RemoveContainer" containerID="2040ddfe522bcc13bab4f250931c8d0f4d886d989e04cf54c7f40aa54bb9b1a1" Nov 03 21:30:05.368051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:05.368031 4963 scope.go:122] "RemoveContainer" containerID="4d956d1d30a528782e9c4699d179b08f847aa7dfa40ef1bde04300cdbbed49e3" Nov 03 21:30:05.368346 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:30:05.368323 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk_odh-ai-gateway-infra(726fefeb-77f7-49bd-b9b0-c4c4425ab096)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" podUID="726fefeb-77f7-49bd-b9b0-c4c4425ab096" Nov 03 21:30:05.371105 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:05.371081 4963 generic.go:356] "Generic (PLEG): container finished" podID="169f11b5-0e68-4242-8fee-82d85da91b3b" containerID="a09f37edc46df77d7ff26cf0c24fc894dc62c3acb59f17c1c3412f342d3bd2e8" exitCode=60 Nov 03 21:30:05.371227 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:05.371169 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" event={"ID":"169f11b5-0e68-4242-8fee-82d85da91b3b","Type":"ContainerDied","Data":"a09f37edc46df77d7ff26cf0c24fc894dc62c3acb59f17c1c3412f342d3bd2e8"} Nov 03 21:30:05.371427 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:05.371412 4963 scope.go:122] "RemoveContainer" containerID="a09f37edc46df77d7ff26cf0c24fc894dc62c3acb59f17c1c3412f342d3bd2e8" Nov 03 21:30:05.371794 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:30:05.371771 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr_odh-ai-gateway-infra(169f11b5-0e68-4242-8fee-82d85da91b3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" podUID="169f11b5-0e68-4242-8fee-82d85da91b3b" Nov 03 21:30:05.373443 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:05.373420 4963 scope.go:122] "RemoveContainer" containerID="1e677fd7f2f2dea3c07eada0ff272769a5ca567ef83cdf51e8d36f2773a494de" Nov 03 21:30:05.373763 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:30:05.373740 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg_odh-ai-gateway-infra(93239ed2-3967-4b97-b37e-6993cad96297)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" podUID="93239ed2-3967-4b97-b37e-6993cad96297" Nov 03 21:30:05.382891 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:05.382860 4963 scope.go:122] "RemoveContainer" containerID="50875337ce952800b81e632a19c794fbd20012f1a9dd9e4252449d543918ae16" Nov 03 21:30:05.766334 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:05.766287 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:06.476924 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:06.469842 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:06.476924 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:06.475606 4963 scope.go:122] "RemoveContainer" containerID="4d956d1d30a528782e9c4699d179b08f847aa7dfa40ef1bde04300cdbbed49e3" Nov 03 21:30:06.476924 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:30:06.475895 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk_odh-ai-gateway-infra(726fefeb-77f7-49bd-b9b0-c4c4425ab096)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" podUID="726fefeb-77f7-49bd-b9b0-c4c4425ab096" Nov 03 21:30:06.481757 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:06.479997 4963 scope.go:122] "RemoveContainer" containerID="a09f37edc46df77d7ff26cf0c24fc894dc62c3acb59f17c1c3412f342d3bd2e8" Nov 03 21:30:06.481757 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:30:06.480342 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr_odh-ai-gateway-infra(169f11b5-0e68-4242-8fee-82d85da91b3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" podUID="169f11b5-0e68-4242-8fee-82d85da91b3b" Nov 03 21:30:06.819143 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:06.819056 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:30:06.819426 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:06.819398 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" podUID="7b117e23-def9-49c7-a6bd-db4ccb5c1a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:30:06.856230 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:06.856189 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:06.995504 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:06.995464 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:30:06.995936 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:06.995877 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:30:06.996709 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:06.996688 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:30:07.054972 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:07.054938 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:08.758089 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:08.758046 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:08.944943 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:08.944891 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:09.193678 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.193635 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v"] Nov 03 21:30:09.199224 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.199200 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.204585 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.204559 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.204733 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.204593 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1650f263-d416-4ba6-8067-63828b7c64d3-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.204733 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.204620 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.204815 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.204795 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.204850 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.204828 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q29w\" (UniqueName: \"kubernetes.io/projected/1650f263-d416-4ba6-8067-63828b7c64d3-kube-api-access-2q29w\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.204884 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.204856 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1650f263-d416-4ba6-8067-63828b7c64d3-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.204939 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.204886 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.205007 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.204992 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.205094 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.205079 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1650f263-d416-4ba6-8067-63828b7c64d3-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.301077 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.301035 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v"] Nov 03 21:30:09.306056 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.306025 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.306227 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.306064 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1650f263-d416-4ba6-8067-63828b7c64d3-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.306227 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.306088 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.306336 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.306212 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.306336 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.306258 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q29w\" (UniqueName: \"kubernetes.io/projected/1650f263-d416-4ba6-8067-63828b7c64d3-kube-api-access-2q29w\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.306336 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.306301 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1650f263-d416-4ba6-8067-63828b7c64d3-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.306502 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.306351 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.306502 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.306390 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.306502 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.306437 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.306675 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.306528 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1650f263-d416-4ba6-8067-63828b7c64d3-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.306743 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.306721 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.306801 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.306752 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.306860 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.306838 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.307351 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.307321 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1650f263-d416-4ba6-8067-63828b7c64d3-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.308535 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.308512 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.308804 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.308784 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1650f263-d416-4ba6-8067-63828b7c64d3-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.364697 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.364662 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1650f263-d416-4ba6-8067-63828b7c64d3-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.364854 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.364779 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q29w\" (UniqueName: \"kubernetes.io/projected/1650f263-d416-4ba6-8067-63828b7c64d3-kube-api-access-2q29w\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.513394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.513299 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:09.945078 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.945039 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v"] Nov 03 21:30:09.950234 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:30:09.950197 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1650f263_d416_4ba6_8067_63828b7c64d3.slice/crio-17e1fa1d86fbc21275611138bf96aa9adb1320e5dffd63358e1246e67b06f7da WatchSource:0}: Error finding container 17e1fa1d86fbc21275611138bf96aa9adb1320e5dffd63358e1246e67b06f7da: Status 404 returned error can't find the container with id 17e1fa1d86fbc21275611138bf96aa9adb1320e5dffd63358e1246e67b06f7da Nov 03 21:30:09.953455 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.953418 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:30:09.953560 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.953512 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:30:09.953560 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:09.953552 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:30:10.510129 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:10.510083 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" event={"ID":"1650f263-d416-4ba6-8067-63828b7c64d3","Type":"ContainerStarted","Data":"da3c6454c3d7a5efabccee7b9189ab13ca04082e0cb15ca672ea225b58a57e3d"} Nov 03 21:30:10.510343 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:10.510152 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" event={"ID":"1650f263-d416-4ba6-8067-63828b7c64d3","Type":"ContainerStarted","Data":"17e1fa1d86fbc21275611138bf96aa9adb1320e5dffd63358e1246e67b06f7da"} Nov 03 21:30:10.513985 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:10.513962 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:10.721227 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:10.721173 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" podStartSLOduration=1.721158316 podStartE2EDuration="1.721158316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:10.671565254 +0000 UTC m=+2559.831637525" watchObservedRunningTime="2026-11-03 21:30:10.721158316 +0000 UTC m=+2559.881230589" Nov 03 21:30:11.514827 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:11.514790 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 21:30:11.515333 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:11.514845 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" podUID="1650f263-d416-4ba6-8067-63828b7c64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 21:30:11.564814 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:11.564769 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:11.761132 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:11.761070 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:11.947684 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:11.947652 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:12.513967 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.513893 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 21:30:12.514142 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.513988 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" podUID="1650f263-d416-4ba6-8067-63828b7c64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 21:30:12.535032 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:30:12.534997 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:30:12.572729 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.572696 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8"] Nov 03 21:30:12.576964 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.576938 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.637016 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.636975 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.637215 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.637025 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.637215 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.637102 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.637215 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.637129 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.637215 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.637150 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.637422 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.637213 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.637422 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.637256 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.637422 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.637369 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.637422 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.637395 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k6xz\" (UniqueName: \"kubernetes.io/projected/bb7fbcac-f9dc-4c3c-beba-0582649b2226-kube-api-access-5k6xz\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.674378 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.674334 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8"] Nov 03 21:30:12.738040 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.737995 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.738040 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.738040 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k6xz\" (UniqueName: \"kubernetes.io/projected/bb7fbcac-f9dc-4c3c-beba-0582649b2226-kube-api-access-5k6xz\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.738315 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.738117 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.738315 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.738152 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.738315 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.738183 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.738315 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.738197 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.738315 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.738226 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.738315 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.738270 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.738315 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.738296 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.738717 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.738696 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.738871 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.738848 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.738954 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.738886 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.739027 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.738977 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.739326 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.739298 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.740567 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.740540 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.741051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.741026 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.808104 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.808001 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k6xz\" (UniqueName: \"kubernetes.io/projected/bb7fbcac-f9dc-4c3c-beba-0582649b2226-kube-api-access-5k6xz\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.808246 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.808140 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:12.917341 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:12.917305 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:13.349820 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:13.349778 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8"] Nov 03 21:30:13.354461 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:30:13.354432 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7fbcac_f9dc_4c3c_beba_0582649b2226.slice/crio-fa87cbd1c5fb18e4d285529abb4742d699e89d6a3efb2f090784952cfdb27316 WatchSource:0}: Error finding container fa87cbd1c5fb18e4d285529abb4742d699e89d6a3efb2f090784952cfdb27316: Status 404 returned error can't find the container with id fa87cbd1c5fb18e4d285529abb4742d699e89d6a3efb2f090784952cfdb27316 Nov 03 21:30:13.357055 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:13.357019 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:30:13.357159 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:13.357090 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:30:13.357159 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:13.357124 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:30:13.513833 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:13.513783 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 21:30:13.514071 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:13.514023 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" podUID="1650f263-d416-4ba6-8067-63828b7c64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 21:30:13.531781 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:13.531745 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" event={"ID":"bb7fbcac-f9dc-4c3c-beba-0582649b2226","Type":"ContainerStarted","Data":"0517b8b1314bfdeedb7333f0dc6d6c8ad2fb23c575d8140f6e4eb3c2dbedd3b0"} Nov 03 21:30:13.531972 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:13.531811 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" event={"ID":"bb7fbcac-f9dc-4c3c-beba-0582649b2226","Type":"ContainerStarted","Data":"fa87cbd1c5fb18e4d285529abb4742d699e89d6a3efb2f090784952cfdb27316"} Nov 03 21:30:13.651008 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:13.650886 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:13.673757 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:13.673692 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" podStartSLOduration=1.6736756480000001 podStartE2EDuration="1.673675648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:13.671768371 +0000 UTC m=+2562.831840646" watchObservedRunningTime="2026-11-03 21:30:13.673675648 +0000 UTC m=+2562.833747920" Nov 03 21:30:13.838399 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:13.838361 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:13.917533 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:13.917440 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:14.523929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:14.523326 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 21:30:14.523929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:14.523386 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" podUID="1650f263-d416-4ba6-8067-63828b7c64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 21:30:14.655463 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:14.655431 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:14.846531 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:14.846426 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:14.930930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:14.930786 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:30:14.931144 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:14.930954 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" podUID="bb7fbcac-f9dc-4c3c-beba-0582649b2226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 21:30:15.525930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:15.523031 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 21:30:15.525930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:15.523098 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" podUID="1650f263-d416-4ba6-8067-63828b7c64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 21:30:15.919987 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:15.919333 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:30:15.919987 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:15.919397 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" podUID="bb7fbcac-f9dc-4c3c-beba-0582649b2226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 21:30:16.513896 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:16.513860 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 21:30:16.514123 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:16.513944 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" podUID="1650f263-d416-4ba6-8067-63828b7c64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 21:30:16.917810 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:16.917720 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:30:16.917810 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:16.917781 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" podUID="bb7fbcac-f9dc-4c3c-beba-0582649b2226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 21:30:17.172478 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:17.172382 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:17.363191 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:17.363128 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:17.694366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:17.694334 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:17.717071 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:17.696174 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:17.717425 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:17.717399 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:30:17.918520 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:17.918475 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:30:17.918697 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:17.918545 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" podUID="bb7fbcac-f9dc-4c3c-beba-0582649b2226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 21:30:17.960826 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:17.960751 4963 scope.go:122] "RemoveContainer" containerID="4d956d1d30a528782e9c4699d179b08f847aa7dfa40ef1bde04300cdbbed49e3" Nov 03 21:30:17.960993 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:17.960874 4963 scope.go:122] "RemoveContainer" containerID="1e677fd7f2f2dea3c07eada0ff272769a5ca567ef83cdf51e8d36f2773a494de" Nov 03 21:30:18.568735 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:18.568699 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" event={"ID":"726fefeb-77f7-49bd-b9b0-c4c4425ab096","Type":"ContainerStarted","Data":"9feca7b34bae115c468e0f8e1d794ff29bc2117aca7164ca7adcb8b19e3fe973"} Nov 03 21:30:18.570701 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:18.570677 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" event={"ID":"93239ed2-3967-4b97-b37e-6993cad96297","Type":"ContainerStarted","Data":"49989694024a6c747679813af873a7d2eb65f2ad15036b68ff3578e871ff3fd9"} Nov 03 21:30:18.918490 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:18.918410 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:30:18.918490 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:18.918471 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" podUID="bb7fbcac-f9dc-4c3c-beba-0582649b2226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 21:30:18.931960 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:18.931926 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:18.960503 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:18.960471 4963 scope.go:122] "RemoveContainer" containerID="a09f37edc46df77d7ff26cf0c24fc894dc62c3acb59f17c1c3412f342d3bd2e8" Nov 03 21:30:19.269522 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:19.269483 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:19.453021 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:19.452989 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:19.578921 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:19.578877 4963 generic.go:356] "Generic (PLEG): container finished" podID="726fefeb-77f7-49bd-b9b0-c4c4425ab096" containerID="9feca7b34bae115c468e0f8e1d794ff29bc2117aca7164ca7adcb8b19e3fe973" exitCode=60 Nov 03 21:30:19.579365 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:19.578931 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" event={"ID":"726fefeb-77f7-49bd-b9b0-c4c4425ab096","Type":"ContainerDied","Data":"9feca7b34bae115c468e0f8e1d794ff29bc2117aca7164ca7adcb8b19e3fe973"} Nov 03 21:30:19.579365 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:19.578991 4963 scope.go:122] "RemoveContainer" containerID="4d956d1d30a528782e9c4699d179b08f847aa7dfa40ef1bde04300cdbbed49e3" Nov 03 21:30:19.579447 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:19.579412 4963 scope.go:122] "RemoveContainer" containerID="9feca7b34bae115c468e0f8e1d794ff29bc2117aca7164ca7adcb8b19e3fe973" Nov 03 21:30:19.579709 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:30:19.579690 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk_odh-ai-gateway-infra(726fefeb-77f7-49bd-b9b0-c4c4425ab096)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" podUID="726fefeb-77f7-49bd-b9b0-c4c4425ab096" Nov 03 21:30:19.589419 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:19.589367 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" event={"ID":"169f11b5-0e68-4242-8fee-82d85da91b3b","Type":"ContainerStarted","Data":"fd8534ae02601127c545599a0277e6f4effb0912041034835778ac99776e6309"} Nov 03 21:30:19.591802 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:19.591772 4963 generic.go:356] "Generic (PLEG): container finished" podID="93239ed2-3967-4b97-b37e-6993cad96297" containerID="49989694024a6c747679813af873a7d2eb65f2ad15036b68ff3578e871ff3fd9" exitCode=60 Nov 03 21:30:19.592021 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:19.591822 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" event={"ID":"93239ed2-3967-4b97-b37e-6993cad96297","Type":"ContainerDied","Data":"49989694024a6c747679813af873a7d2eb65f2ad15036b68ff3578e871ff3fd9"} Nov 03 21:30:19.592469 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:19.592450 4963 scope.go:122] "RemoveContainer" containerID="49989694024a6c747679813af873a7d2eb65f2ad15036b68ff3578e871ff3fd9" Nov 03 21:30:19.592698 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:30:19.592680 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg_odh-ai-gateway-infra(93239ed2-3967-4b97-b37e-6993cad96297)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" podUID="93239ed2-3967-4b97-b37e-6993cad96297" Nov 03 21:30:19.600415 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:19.600393 4963 scope.go:122] "RemoveContainer" containerID="1e677fd7f2f2dea3c07eada0ff272769a5ca567ef83cdf51e8d36f2773a494de" Nov 03 21:30:19.688579 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:19.688543 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk"] Nov 03 21:30:19.801479 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:19.801382 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg"] Nov 03 21:30:19.917963 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:19.917880 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:30:19.918127 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:19.917979 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" podUID="bb7fbcac-f9dc-4c3c-beba-0582649b2226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 21:30:20.604214 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:20.604177 4963 generic.go:356] "Generic (PLEG): container finished" podID="169f11b5-0e68-4242-8fee-82d85da91b3b" containerID="fd8534ae02601127c545599a0277e6f4effb0912041034835778ac99776e6309" exitCode=60 Nov 03 21:30:20.604686 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:20.604276 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" event={"ID":"169f11b5-0e68-4242-8fee-82d85da91b3b","Type":"ContainerDied","Data":"fd8534ae02601127c545599a0277e6f4effb0912041034835778ac99776e6309"} Nov 03 21:30:20.604686 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:20.604351 4963 scope.go:122] "RemoveContainer" containerID="a09f37edc46df77d7ff26cf0c24fc894dc62c3acb59f17c1c3412f342d3bd2e8" Nov 03 21:30:20.604802 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:20.604747 4963 scope.go:122] "RemoveContainer" containerID="fd8534ae02601127c545599a0277e6f4effb0912041034835778ac99776e6309" Nov 03 21:30:20.605153 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:30:20.605116 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr_odh-ai-gateway-infra(169f11b5-0e68-4242-8fee-82d85da91b3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" podUID="169f11b5-0e68-4242-8fee-82d85da91b3b" Nov 03 21:30:20.858172 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:20.858095 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" Nov 03 21:30:20.860503 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:20.860474 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" Nov 03 21:30:20.886688 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:20.886652 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/726fefeb-77f7-49bd-b9b0-c4c4425ab096-kube-api-access-66smt\" (UniqueName: \"kubernetes.io/projected/726fefeb-77f7-49bd-b9b0-c4c4425ab096-kube-api-access-66smt\") pod \"726fefeb-77f7-49bd-b9b0-c4c4425ab096\" (UID: \"726fefeb-77f7-49bd-b9b0-c4c4425ab096\") " Nov 03 21:30:20.886892 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:20.886810 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/726fefeb-77f7-49bd-b9b0-c4c4425ab096-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/726fefeb-77f7-49bd-b9b0-c4c4425ab096-maas-api-service-ca\") pod \"726fefeb-77f7-49bd-b9b0-c4c4425ab096\" (UID: \"726fefeb-77f7-49bd-b9b0-c4c4425ab096\") " Nov 03 21:30:20.887205 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:20.887180 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726fefeb-77f7-49bd-b9b0-c4c4425ab096-maas-api-service-ca" pod "726fefeb-77f7-49bd-b9b0-c4c4425ab096" (UID: "726fefeb-77f7-49bd-b9b0-c4c4425ab096"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:20.889032 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:20.889002 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726fefeb-77f7-49bd-b9b0-c4c4425ab096-kube-api-access-66smt" pod "726fefeb-77f7-49bd-b9b0-c4c4425ab096" (UID: "726fefeb-77f7-49bd-b9b0-c4c4425ab096"). InnerVolumeSpecName "kube-api-access-66smt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:20.929920 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:20.929864 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr"] Nov 03 21:30:20.988058 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:20.988025 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93239ed2-3967-4b97-b37e-6993cad96297-kube-api-access-5nqmv\" (UniqueName: \"kubernetes.io/projected/93239ed2-3967-4b97-b37e-6993cad96297-kube-api-access-5nqmv\") pod \"93239ed2-3967-4b97-b37e-6993cad96297\" (UID: \"93239ed2-3967-4b97-b37e-6993cad96297\") " Nov 03 21:30:20.988219 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:20.988076 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93239ed2-3967-4b97-b37e-6993cad96297-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93239ed2-3967-4b97-b37e-6993cad96297-maas-api-service-ca\") pod \"93239ed2-3967-4b97-b37e-6993cad96297\" (UID: \"93239ed2-3967-4b97-b37e-6993cad96297\") " Nov 03 21:30:20.988519 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:20.988490 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93239ed2-3967-4b97-b37e-6993cad96297-maas-api-service-ca" pod "93239ed2-3967-4b97-b37e-6993cad96297" (UID: "93239ed2-3967-4b97-b37e-6993cad96297"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:20.988685 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:20.988665 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66smt\" (UniqueName: \"kubernetes.io/projected/726fefeb-77f7-49bd-b9b0-c4c4425ab096-kube-api-access-66smt\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:20.988725 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:20.988692 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/726fefeb-77f7-49bd-b9b0-c4c4425ab096-maas-api-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:20.988725 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:20.988709 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93239ed2-3967-4b97-b37e-6993cad96297-maas-api-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:20.990312 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:20.990289 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93239ed2-3967-4b97-b37e-6993cad96297-kube-api-access-5nqmv" pod "93239ed2-3967-4b97-b37e-6993cad96297" (UID: "93239ed2-3967-4b97-b37e-6993cad96297"). InnerVolumeSpecName "kube-api-access-5nqmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:21.062887 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.062856 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:21.063177 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.063156 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:21.064181 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.064162 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:30:21.089867 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.089830 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nqmv\" (UniqueName: \"kubernetes.io/projected/93239ed2-3967-4b97-b37e-6993cad96297-kube-api-access-5nqmv\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:21.614450 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.614418 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" Nov 03 21:30:21.614450 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.614433 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk" event={"ID":"726fefeb-77f7-49bd-b9b0-c4c4425ab096","Type":"ContainerDied","Data":"82d33a3a23acd9037af5ccb2ad74c42006823df59fec434b69b2e9e7ed424da8"} Nov 03 21:30:21.615038 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.614505 4963 scope.go:122] "RemoveContainer" containerID="9feca7b34bae115c468e0f8e1d794ff29bc2117aca7164ca7adcb8b19e3fe973" Nov 03 21:30:21.618083 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.618057 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" Nov 03 21:30:21.618178 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.618121 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg" event={"ID":"93239ed2-3967-4b97-b37e-6993cad96297","Type":"ContainerDied","Data":"6c3f85223f9d04247615b70a8d26abe5bd9fcea689cce19c9300afc24fd0830d"} Nov 03 21:30:21.624621 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.624600 4963 scope.go:122] "RemoveContainer" containerID="49989694024a6c747679813af873a7d2eb65f2ad15036b68ff3578e871ff3fd9" Nov 03 21:30:21.751406 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.751378 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" Nov 03 21:30:21.770886 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.770852 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg"] Nov 03 21:30:21.775165 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.775136 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3d317d-29895690-f8fzg"] Nov 03 21:30:21.798695 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.798660 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169f11b5-0e68-4242-8fee-82d85da91b3b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169f11b5-0e68-4242-8fee-82d85da91b3b-maas-api-service-ca\") pod \"169f11b5-0e68-4242-8fee-82d85da91b3b\" (UID: \"169f11b5-0e68-4242-8fee-82d85da91b3b\") " Nov 03 21:30:21.798845 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.798815 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169f11b5-0e68-4242-8fee-82d85da91b3b-kube-api-access-l5xkt\" (UniqueName: \"kubernetes.io/projected/169f11b5-0e68-4242-8fee-82d85da91b3b-kube-api-access-l5xkt\") pod \"169f11b5-0e68-4242-8fee-82d85da91b3b\" (UID: \"169f11b5-0e68-4242-8fee-82d85da91b3b\") " Nov 03 21:30:21.799120 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.799054 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169f11b5-0e68-4242-8fee-82d85da91b3b-maas-api-service-ca" pod "169f11b5-0e68-4242-8fee-82d85da91b3b" (UID: "169f11b5-0e68-4242-8fee-82d85da91b3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:21.799314 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.799286 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169f11b5-0e68-4242-8fee-82d85da91b3b-maas-api-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:21.801095 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.801072 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169f11b5-0e68-4242-8fee-82d85da91b3b-kube-api-access-l5xkt" pod "169f11b5-0e68-4242-8fee-82d85da91b3b" (UID: "169f11b5-0e68-4242-8fee-82d85da91b3b"). InnerVolumeSpecName "kube-api-access-l5xkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:21.900678 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.900592 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5xkt\" (UniqueName: \"kubernetes.io/projected/169f11b5-0e68-4242-8fee-82d85da91b3b-kube-api-access-l5xkt\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:21.990022 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.989988 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk"] Nov 03 21:30:21.994004 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:21.993967 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895690-t59fk"] Nov 03 21:30:22.626650 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:22.626616 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" Nov 03 21:30:22.627144 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:22.626619 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr" event={"ID":"169f11b5-0e68-4242-8fee-82d85da91b3b","Type":"ContainerDied","Data":"7e3bf8dec7af75df806faa6544f69e9c4bc0f2e3f422e0154d626d2fc82c27cf"} Nov 03 21:30:22.627144 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:22.626759 4963 scope.go:122] "RemoveContainer" containerID="fd8534ae02601127c545599a0277e6f4effb0912041034835778ac99776e6309" Nov 03 21:30:22.783150 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:22.783113 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr"] Nov 03 21:30:22.830437 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:22.830399 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895690-69xvr"] Nov 03 21:30:22.968625 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:22.968587 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="169f11b5-0e68-4242-8fee-82d85da91b3b" path="/var/lib/kubelet/pods/169f11b5-0e68-4242-8fee-82d85da91b3b/volumes" Nov 03 21:30:22.969170 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:22.969148 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="726fefeb-77f7-49bd-b9b0-c4c4425ab096" path="/var/lib/kubelet/pods/726fefeb-77f7-49bd-b9b0-c4c4425ab096/volumes" Nov 03 21:30:22.969640 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:22.969621 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93239ed2-3967-4b97-b37e-6993cad96297" path="/var/lib/kubelet/pods/93239ed2-3967-4b97-b37e-6993cad96297/volumes" Nov 03 21:30:23.758556 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:23.758522 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:23.947144 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:23.947106 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:24.819673 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:24.819633 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:25.499383 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:25.499335 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm"] Nov 03 21:30:25.577151 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:25.577120 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:25.606319 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:25.606279 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z"] Nov 03 21:30:25.908159 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:25.908070 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:26.102049 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:26.102003 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:28.640653 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:28.640610 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:30.500612 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:30.500539 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" podUID="6e9e38ae-d8ba-402e-bda3-312acd372dec" containerName="payload-processing" containerID="cri-o://b54cccbfcc8b6a579ea160b8e55f2c82fad7d9cdb010e2a6a393a2ed5927208b" gracePeriod=25 Nov 03 21:30:30.607586 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:30.607498 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" podUID="db4b1b6e-bc5a-4854-97df-e14361e3f643" containerName="payload-pre-processing" containerID="cri-o://e765734b2f7d9550911de5a44355baa7dad05251ea33d50ec52ad2ae421da081" gracePeriod=25 Nov 03 21:30:31.381065 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:31.381027 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:33.171468 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.171431 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs"] Nov 03 21:30:33.172070 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.172051 4963 cpu_manager.go:414] "containerMap: removing container" podUID="169f11b5-0e68-4242-8fee-82d85da91b3b" containerName="cleanup" Nov 03 21:30:33.172126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.172071 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169f11b5-0e68-4242-8fee-82d85da91b3b" containerName="cleanup" Nov 03 21:30:33.172126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.172081 4963 cpu_manager.go:414] "containerMap: removing container" podUID="726fefeb-77f7-49bd-b9b0-c4c4425ab096" containerName="cleanup" Nov 03 21:30:33.172126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.172085 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="726fefeb-77f7-49bd-b9b0-c4c4425ab096" containerName="cleanup" Nov 03 21:30:33.172126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.172115 4963 cpu_manager.go:414] "containerMap: removing container" podUID="93239ed2-3967-4b97-b37e-6993cad96297" containerName="cleanup" Nov 03 21:30:33.172126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.172121 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93239ed2-3967-4b97-b37e-6993cad96297" containerName="cleanup" Nov 03 21:30:33.172298 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.172130 4963 cpu_manager.go:414] "containerMap: removing container" podUID="93239ed2-3967-4b97-b37e-6993cad96297" containerName="cleanup" Nov 03 21:30:33.172298 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.172135 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93239ed2-3967-4b97-b37e-6993cad96297" containerName="cleanup" Nov 03 21:30:33.172298 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.172152 4963 cpu_manager.go:414] "containerMap: removing container" podUID="726fefeb-77f7-49bd-b9b0-c4c4425ab096" containerName="cleanup" Nov 03 21:30:33.172298 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.172157 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="726fefeb-77f7-49bd-b9b0-c4c4425ab096" containerName="cleanup" Nov 03 21:30:33.172298 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.172169 4963 cpu_manager.go:414] "containerMap: removing container" podUID="169f11b5-0e68-4242-8fee-82d85da91b3b" containerName="cleanup" Nov 03 21:30:33.172298 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.172175 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169f11b5-0e68-4242-8fee-82d85da91b3b" containerName="cleanup" Nov 03 21:30:33.172298 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.172260 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="726fefeb-77f7-49bd-b9b0-c4c4425ab096" containerName="cleanup" Nov 03 21:30:33.172298 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.172268 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="93239ed2-3967-4b97-b37e-6993cad96297" containerName="cleanup" Nov 03 21:30:33.172298 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.172274 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="169f11b5-0e68-4242-8fee-82d85da91b3b" containerName="cleanup" Nov 03 21:30:33.172298 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.172281 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="169f11b5-0e68-4242-8fee-82d85da91b3b" containerName="cleanup" Nov 03 21:30:33.172298 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.172285 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="726fefeb-77f7-49bd-b9b0-c4c4425ab096" containerName="cleanup" Nov 03 21:30:33.172298 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.172295 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="93239ed2-3967-4b97-b37e-6993cad96297" containerName="cleanup" Nov 03 21:30:33.177542 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.177518 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 21:30:33.228277 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.228148 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-9818d8-serving-cert\"" Nov 03 21:30:33.274809 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.274767 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e33113d6-3645-47a5-8f09-edee50ffb54d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs\" (UID: \"e33113d6-3645-47a5-8f09-edee50ffb54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 21:30:33.275013 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.274822 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e33113d6-3645-47a5-8f09-edee50ffb54d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs\" (UID: \"e33113d6-3645-47a5-8f09-edee50ffb54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 21:30:33.275013 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.274958 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e33113d6-3645-47a5-8f09-edee50ffb54d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs\" (UID: \"e33113d6-3645-47a5-8f09-edee50ffb54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 21:30:33.275098 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.275028 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfj2k\" (UniqueName: \"kubernetes.io/projected/e33113d6-3645-47a5-8f09-edee50ffb54d-kube-api-access-xfj2k\") pod \"maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs\" (UID: \"e33113d6-3645-47a5-8f09-edee50ffb54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 21:30:33.278085 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.278060 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs"] Nov 03 21:30:33.376300 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.376263 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e33113d6-3645-47a5-8f09-edee50ffb54d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs\" (UID: \"e33113d6-3645-47a5-8f09-edee50ffb54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 21:30:33.376441 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.376349 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfj2k\" (UniqueName: \"kubernetes.io/projected/e33113d6-3645-47a5-8f09-edee50ffb54d-kube-api-access-xfj2k\") pod \"maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs\" (UID: \"e33113d6-3645-47a5-8f09-edee50ffb54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 21:30:33.376441 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:30:33.376410 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-serving-cert: secret "maas-api-e2e-shared-a-w2-9818d8-serving-cert" not found Nov 03 21:30:33.376514 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:30:33.376508 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e33113d6-3645-47a5-8f09-edee50ffb54d-maas-api-tls podName:e33113d6-3645-47a5-8f09-edee50ffb54d nodeName:}" failed. No retries permitted until 2026-11-03 21:30:33.876479817 +0000 UTC m=+2583.036552069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e33113d6-3645-47a5-8f09-edee50ffb54d-maas-api-tls") pod "maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" (UID: "e33113d6-3645-47a5-8f09-edee50ffb54d") : secret "maas-api-e2e-shared-a-w2-9818d8-serving-cert" not found Nov 03 21:30:33.376667 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.376636 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e33113d6-3645-47a5-8f09-edee50ffb54d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs\" (UID: \"e33113d6-3645-47a5-8f09-edee50ffb54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 21:30:33.376758 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.376691 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e33113d6-3645-47a5-8f09-edee50ffb54d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs\" (UID: \"e33113d6-3645-47a5-8f09-edee50ffb54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 21:30:33.377338 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.377315 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e33113d6-3645-47a5-8f09-edee50ffb54d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs\" (UID: \"e33113d6-3645-47a5-8f09-edee50ffb54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 21:30:33.379347 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.379319 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e33113d6-3645-47a5-8f09-edee50ffb54d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs\" (UID: \"e33113d6-3645-47a5-8f09-edee50ffb54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 21:30:33.435641 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.435568 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfj2k\" (UniqueName: \"kubernetes.io/projected/e33113d6-3645-47a5-8f09-edee50ffb54d-kube-api-access-xfj2k\") pod \"maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs\" (UID: \"e33113d6-3645-47a5-8f09-edee50ffb54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 21:30:33.512051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.512015 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:33.722181 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.722146 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:33.884844 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.884798 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e33113d6-3645-47a5-8f09-edee50ffb54d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs\" (UID: \"e33113d6-3645-47a5-8f09-edee50ffb54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 21:30:33.887462 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:33.887432 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e33113d6-3645-47a5-8f09-edee50ffb54d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs\" (UID: \"e33113d6-3645-47a5-8f09-edee50ffb54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 21:30:34.092654 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:34.092567 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 21:30:34.167077 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:34.167032 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" podUID="db4b1b6e-bc5a-4854-97df-e14361e3f643" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:30:34.167427 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:34.167396 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" podUID="6e9e38ae-d8ba-402e-bda3-312acd372dec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:30:34.469528 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:34.469489 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs"] Nov 03 21:30:34.473803 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:30:34.473777 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33113d6_3645_47a5_8f09_edee50ffb54d.slice/crio-ddc9a6cdc07c47e834867987e6783dba0278751183cf5064c9c8a97f9ffe94d1 WatchSource:0}: Error finding container ddc9a6cdc07c47e834867987e6783dba0278751183cf5064c9c8a97f9ffe94d1: Status 404 returned error can't find the container with id ddc9a6cdc07c47e834867987e6783dba0278751183cf5064c9c8a97f9ffe94d1 Nov 03 21:30:34.475707 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:34.475689 4963 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:30:34.712916 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:34.712866 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" event={"ID":"e33113d6-3645-47a5-8f09-edee50ffb54d","Type":"ContainerStarted","Data":"ddc9a6cdc07c47e834867987e6783dba0278751183cf5064c9c8a97f9ffe94d1"} Nov 03 21:30:35.521503 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:35.521411 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:35.720743 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:35.720699 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" event={"ID":"e33113d6-3645-47a5-8f09-edee50ffb54d","Type":"ContainerStarted","Data":"bf251979acefd9730c0ed879339e05aa2607a4a4a43aa94b4addb9911e71af38"} Nov 03 21:30:35.720924 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:35.720761 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 21:30:35.834069 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:35.834008 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" podStartSLOduration=1.978606698 podStartE2EDuration="2.833994049s" totalImagesPullingTime="855.387351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:35.832066498 +0000 UTC m=+2584.992138773" watchObservedRunningTime="2026-11-03 21:30:35.833994049 +0000 UTC m=+2584.994066320" Nov 03 21:30:37.343765 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:37.343727 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:38.431333 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:38.431292 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:41.733631 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:41.733604 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 21:30:42.312244 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:42.312197 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:42.530135 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:30:42.530089 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:30:43.937601 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:43.937565 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:44.167420 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:44.167380 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" podUID="db4b1b6e-bc5a-4854-97df-e14361e3f643" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:30:44.167780 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:44.167755 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" podUID="6e9e38ae-d8ba-402e-bda3-312acd372dec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:30:48.430263 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:48.430223 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:50.785456 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:50.785429 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" Nov 03 21:30:50.822626 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:50.822590 4963 generic.go:356] "Generic (PLEG): container finished" podID="6e9e38ae-d8ba-402e-bda3-312acd372dec" containerID="b54cccbfcc8b6a579ea160b8e55f2c82fad7d9cdb010e2a6a393a2ed5927208b" exitCode=0 Nov 03 21:30:50.822823 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:50.822642 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" Nov 03 21:30:50.822823 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:50.822670 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" event={"ID":"6e9e38ae-d8ba-402e-bda3-312acd372dec","Type":"ContainerDied","Data":"b54cccbfcc8b6a579ea160b8e55f2c82fad7d9cdb010e2a6a393a2ed5927208b"} Nov 03 21:30:50.822823 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:50.822737 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm" event={"ID":"6e9e38ae-d8ba-402e-bda3-312acd372dec","Type":"ContainerDied","Data":"3c2a9f4e2e72bee5e9abf9edd55d6d01153cbd5cb35e9ab8044b38be23741d35"} Nov 03 21:30:50.822823 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:50.822767 4963 scope.go:122] "RemoveContainer" containerID="b54cccbfcc8b6a579ea160b8e55f2c82fad7d9cdb010e2a6a393a2ed5927208b" Nov 03 21:30:50.824516 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:50.824490 4963 generic.go:356] "Generic (PLEG): container finished" podID="db4b1b6e-bc5a-4854-97df-e14361e3f643" containerID="e765734b2f7d9550911de5a44355baa7dad05251ea33d50ec52ad2ae421da081" exitCode=0 Nov 03 21:30:50.824644 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:50.824557 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" event={"ID":"db4b1b6e-bc5a-4854-97df-e14361e3f643","Type":"ContainerDied","Data":"e765734b2f7d9550911de5a44355baa7dad05251ea33d50ec52ad2ae421da081"} Nov 03 21:30:50.837056 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:50.837019 4963 scope.go:122] "RemoveContainer" containerID="b54cccbfcc8b6a579ea160b8e55f2c82fad7d9cdb010e2a6a393a2ed5927208b" Nov 03 21:30:50.837429 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:30:50.837397 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b54cccbfcc8b6a579ea160b8e55f2c82fad7d9cdb010e2a6a393a2ed5927208b\": container with ID starting with b54cccbfcc8b6a579ea160b8e55f2c82fad7d9cdb010e2a6a393a2ed5927208b not found: ID does not exist" containerID="b54cccbfcc8b6a579ea160b8e55f2c82fad7d9cdb010e2a6a393a2ed5927208b" Nov 03 21:30:50.837537 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:50.837435 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b54cccbfcc8b6a579ea160b8e55f2c82fad7d9cdb010e2a6a393a2ed5927208b"} err="failed to get container status \"b54cccbfcc8b6a579ea160b8e55f2c82fad7d9cdb010e2a6a393a2ed5927208b\": rpc error: code = NotFound desc = could not find container \"b54cccbfcc8b6a579ea160b8e55f2c82fad7d9cdb010e2a6a393a2ed5927208b\": container with ID starting with b54cccbfcc8b6a579ea160b8e55f2c82fad7d9cdb010e2a6a393a2ed5927208b not found: ID does not exist" Nov 03 21:30:50.902361 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:50.902332 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" Nov 03 21:30:50.911370 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:50.911338 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e9e38ae-d8ba-402e-bda3-312acd372dec-kube-api-access-9jfmq\" (UniqueName: \"kubernetes.io/projected/6e9e38ae-d8ba-402e-bda3-312acd372dec-kube-api-access-9jfmq\") pod \"6e9e38ae-d8ba-402e-bda3-312acd372dec\" (UID: \"6e9e38ae-d8ba-402e-bda3-312acd372dec\") " Nov 03 21:30:50.911545 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:50.911458 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e9e38ae-d8ba-402e-bda3-312acd372dec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9e38ae-d8ba-402e-bda3-312acd372dec-plugins-config-volume\") pod \"6e9e38ae-d8ba-402e-bda3-312acd372dec\" (UID: \"6e9e38ae-d8ba-402e-bda3-312acd372dec\") " Nov 03 21:30:50.911893 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:50.911863 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e9e38ae-d8ba-402e-bda3-312acd372dec-plugins-config-volume" pod "6e9e38ae-d8ba-402e-bda3-312acd372dec" (UID: "6e9e38ae-d8ba-402e-bda3-312acd372dec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:50.912043 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:50.912011 4963 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9e38ae-d8ba-402e-bda3-312acd372dec-plugins-config-volume\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:50.914165 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:50.914141 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9e38ae-d8ba-402e-bda3-312acd372dec-kube-api-access-9jfmq" pod "6e9e38ae-d8ba-402e-bda3-312acd372dec" (UID: "6e9e38ae-d8ba-402e-bda3-312acd372dec"). InnerVolumeSpecName "kube-api-access-9jfmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:51.013282 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:51.013201 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4b1b6e-bc5a-4854-97df-e14361e3f643-kube-api-access-57t65\" (UniqueName: \"kubernetes.io/projected/db4b1b6e-bc5a-4854-97df-e14361e3f643-kube-api-access-57t65\") pod \"db4b1b6e-bc5a-4854-97df-e14361e3f643\" (UID: \"db4b1b6e-bc5a-4854-97df-e14361e3f643\") " Nov 03 21:30:51.013439 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:51.013348 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4b1b6e-bc5a-4854-97df-e14361e3f643-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4b1b6e-bc5a-4854-97df-e14361e3f643-plugins-config-volume\") pod \"db4b1b6e-bc5a-4854-97df-e14361e3f643\" (UID: \"db4b1b6e-bc5a-4854-97df-e14361e3f643\") " Nov 03 21:30:51.013693 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:51.013676 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jfmq\" (UniqueName: \"kubernetes.io/projected/6e9e38ae-d8ba-402e-bda3-312acd372dec-kube-api-access-9jfmq\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:51.013810 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:51.013698 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4b1b6e-bc5a-4854-97df-e14361e3f643-plugins-config-volume" pod "db4b1b6e-bc5a-4854-97df-e14361e3f643" (UID: "db4b1b6e-bc5a-4854-97df-e14361e3f643"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:51.015371 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:51.015347 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4b1b6e-bc5a-4854-97df-e14361e3f643-kube-api-access-57t65" pod "db4b1b6e-bc5a-4854-97df-e14361e3f643" (UID: "db4b1b6e-bc5a-4854-97df-e14361e3f643"). InnerVolumeSpecName "kube-api-access-57t65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:51.114968 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:51.114924 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57t65\" (UniqueName: \"kubernetes.io/projected/db4b1b6e-bc5a-4854-97df-e14361e3f643-kube-api-access-57t65\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:51.114968 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:51.114962 4963 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4b1b6e-bc5a-4854-97df-e14361e3f643-plugins-config-volume\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:51.232119 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:51.232027 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm"] Nov 03 21:30:51.238382 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:51.238348 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3d317d-97554d9fb-b97xm"] Nov 03 21:30:51.831787 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:51.831697 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" Nov 03 21:30:51.831787 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:51.831723 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z" event={"ID":"db4b1b6e-bc5a-4854-97df-e14361e3f643","Type":"ContainerDied","Data":"4c5b3147717678f318d8bee754dfff5c5c4a79e4c05fd657b8351ff3042357e2"} Nov 03 21:30:51.832310 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:51.831823 4963 scope.go:122] "RemoveContainer" containerID="e765734b2f7d9550911de5a44355baa7dad05251ea33d50ec52ad2ae421da081" Nov 03 21:30:51.988995 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:51.988959 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z"] Nov 03 21:30:52.036654 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:52.036626 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3d317d-8d9465bdc-k225z"] Nov 03 21:30:52.206789 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:52.206758 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:52.393775 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:52.393743 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:52.967019 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:52.966988 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e9e38ae-d8ba-402e-bda3-312acd372dec" path="/var/lib/kubelet/pods/6e9e38ae-d8ba-402e-bda3-312acd372dec/volumes" Nov 03 21:30:52.967423 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:52.967409 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4b1b6e-bc5a-4854-97df-e14361e3f643" path="/var/lib/kubelet/pods/db4b1b6e-bc5a-4854-97df-e14361e3f643/volumes" Nov 03 21:30:53.048185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:53.048152 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:53.758124 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:53.758082 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:54.272973 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:54.272937 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:55.114186 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:55.114152 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:57.687652 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:57.687623 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:57.875326 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:57.875298 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:58.060014 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:30:58.059969 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:01.134227 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:01.134179 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:01.849575 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:01.849545 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:02.342185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:02.342140 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:02.550533 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:02.550499 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:03.599873 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:03.599839 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:03.801442 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:03.801409 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:03.988791 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:03.988755 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:06.768447 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:06.768416 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:08.400336 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:08.400283 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:12.528025 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:31:12.527982 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:31:12.551680 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:12.551646 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:12.739458 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:12.739429 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:12.921730 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:12.921652 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:14.098335 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:14.098301 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:14.588862 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:14.588831 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:15.433071 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:15.433032 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:15.630156 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:15.630121 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:17.662299 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:17.662264 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:22.884758 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:22.884714 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:26.810055 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:26.810024 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:27.436196 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:27.436160 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:27.622216 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:27.622176 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:27.963921 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:27.963881 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:31.993433 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:31.993389 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:32.309737 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:32.309648 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:32.504185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:32.504153 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:34.607675 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:34.607640 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:35.861825 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:35.861779 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:36.356112 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:36.356080 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:37.304851 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:37.304803 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:38.336222 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:38.336179 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:38.988598 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:38.988562 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:39.187708 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:39.187667 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:39.384579 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:39.384472 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:39.577222 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:39.577183 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:41.919554 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:41.919467 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:42.529114 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:31:42.529076 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:31:44.189615 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:44.189583 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:44.374003 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:44.373973 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:44.996037 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:44.996004 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:45.979186 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:45.979141 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:48.550495 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:48.550450 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:49.565653 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:49.565615 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:49.755770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:49.755742 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:51.346795 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:51.346754 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:51.548754 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:51.548717 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:51.827696 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:51.827650 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:53.387669 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:53.387632 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:54.598995 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:54.598944 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:54.785518 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:54.785474 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:56.551997 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:56.551947 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:31:59.757972 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:31:59.757937 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:00.391522 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:00.391464 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:00.745573 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:00.745521 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:01.451053 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:01.451011 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:04.357329 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:04.357284 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:04.585452 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:04.585400 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:04.784340 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:04.784305 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:05.704917 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:05.704867 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:06.345705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:06.345676 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:06.527268 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:06.527234 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:11.048971 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:11.048938 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:11.900803 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:11.900767 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:12.090980 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.090946 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:12.276729 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.276692 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq"] Nov 03 21:32:12.277221 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.277207 4963 cpu_manager.go:414] "containerMap: removing container" podUID="93239ed2-3967-4b97-b37e-6993cad96297" containerName="cleanup" Nov 03 21:32:12.277221 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.277221 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93239ed2-3967-4b97-b37e-6993cad96297" containerName="cleanup" Nov 03 21:32:12.277366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.277236 4963 cpu_manager.go:414] "containerMap: removing container" podUID="6e9e38ae-d8ba-402e-bda3-312acd372dec" containerName="payload-processing" Nov 03 21:32:12.277366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.277242 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e9e38ae-d8ba-402e-bda3-312acd372dec" containerName="payload-processing" Nov 03 21:32:12.277366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.277256 4963 cpu_manager.go:414] "containerMap: removing container" podUID="726fefeb-77f7-49bd-b9b0-c4c4425ab096" containerName="cleanup" Nov 03 21:32:12.277366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.277261 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="726fefeb-77f7-49bd-b9b0-c4c4425ab096" containerName="cleanup" Nov 03 21:32:12.277366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.277278 4963 cpu_manager.go:414] "containerMap: removing container" podUID="db4b1b6e-bc5a-4854-97df-e14361e3f643" containerName="payload-pre-processing" Nov 03 21:32:12.277366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.277283 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4b1b6e-bc5a-4854-97df-e14361e3f643" containerName="payload-pre-processing" Nov 03 21:32:12.277366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.277290 4963 cpu_manager.go:414] "containerMap: removing container" podUID="169f11b5-0e68-4242-8fee-82d85da91b3b" containerName="cleanup" Nov 03 21:32:12.277366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.277294 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169f11b5-0e68-4242-8fee-82d85da91b3b" containerName="cleanup" Nov 03 21:32:12.277802 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.277386 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e9e38ae-d8ba-402e-bda3-312acd372dec" containerName="payload-processing" Nov 03 21:32:12.277802 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.277395 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="93239ed2-3967-4b97-b37e-6993cad96297" containerName="cleanup" Nov 03 21:32:12.277802 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.277402 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4b1b6e-bc5a-4854-97df-e14361e3f643" containerName="payload-pre-processing" Nov 03 21:32:12.277802 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.277408 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="169f11b5-0e68-4242-8fee-82d85da91b3b" containerName="cleanup" Nov 03 21:32:12.277802 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.277413 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="726fefeb-77f7-49bd-b9b0-c4c4425ab096" containerName="cleanup" Nov 03 21:32:12.281244 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.281216 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq" Nov 03 21:32:12.283630 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.283602 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:12.350350 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.350314 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brflb\" (UniqueName: \"kubernetes.io/projected/9204dab2-ccbe-4405-b9c7-666f23309bbd-kube-api-access-brflb\") pod \"payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq\" (UID: \"9204dab2-ccbe-4405-b9c7-666f23309bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq" Nov 03 21:32:12.350586 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.350564 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9204dab2-ccbe-4405-b9c7-666f23309bbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq\" (UID: \"9204dab2-ccbe-4405-b9c7-666f23309bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq" Nov 03 21:32:12.378949 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.378866 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-f0dc66\"" Nov 03 21:32:12.384764 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.384727 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq"] Nov 03 21:32:12.385015 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.384995 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-f0dc66-dockercfg-n8782\"" Nov 03 21:32:12.432929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.432005 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9"] Nov 03 21:32:12.437938 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.437079 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9" Nov 03 21:32:12.451419 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.451379 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brflb\" (UniqueName: \"kubernetes.io/projected/9204dab2-ccbe-4405-b9c7-666f23309bbd-kube-api-access-brflb\") pod \"payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq\" (UID: \"9204dab2-ccbe-4405-b9c7-666f23309bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq" Nov 03 21:32:12.451609 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.451532 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9204dab2-ccbe-4405-b9c7-666f23309bbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq\" (UID: \"9204dab2-ccbe-4405-b9c7-666f23309bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq" Nov 03 21:32:12.452617 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.452587 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9204dab2-ccbe-4405-b9c7-666f23309bbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq\" (UID: \"9204dab2-ccbe-4405-b9c7-666f23309bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq" Nov 03 21:32:12.482788 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.482743 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9"] Nov 03 21:32:12.516086 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.511646 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brflb\" (UniqueName: \"kubernetes.io/projected/9204dab2-ccbe-4405-b9c7-666f23309bbd-kube-api-access-brflb\") pod \"payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq\" (UID: \"9204dab2-ccbe-4405-b9c7-666f23309bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq" Nov 03 21:32:12.537590 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:32:12.537494 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:32:12.552807 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.552771 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58309a66-dfdf-4793-bbc5-5c7b33ca111e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9\" (UID: \"58309a66-dfdf-4793-bbc5-5c7b33ca111e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9" Nov 03 21:32:12.553030 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.552888 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5n6p\" (UniqueName: \"kubernetes.io/projected/58309a66-dfdf-4793-bbc5-5c7b33ca111e-kube-api-access-j5n6p\") pod \"payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9\" (UID: \"58309a66-dfdf-4793-bbc5-5c7b33ca111e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9" Nov 03 21:32:12.595737 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.595697 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq" Nov 03 21:32:12.654314 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.654266 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58309a66-dfdf-4793-bbc5-5c7b33ca111e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9\" (UID: \"58309a66-dfdf-4793-bbc5-5c7b33ca111e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9" Nov 03 21:32:12.654446 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.654429 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5n6p\" (UniqueName: \"kubernetes.io/projected/58309a66-dfdf-4793-bbc5-5c7b33ca111e-kube-api-access-j5n6p\") pod \"payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9\" (UID: \"58309a66-dfdf-4793-bbc5-5c7b33ca111e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9" Nov 03 21:32:12.655240 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.655207 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58309a66-dfdf-4793-bbc5-5c7b33ca111e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9\" (UID: \"58309a66-dfdf-4793-bbc5-5c7b33ca111e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9" Nov 03 21:32:12.713679 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.713551 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5n6p\" (UniqueName: \"kubernetes.io/projected/58309a66-dfdf-4793-bbc5-5c7b33ca111e-kube-api-access-j5n6p\") pod \"payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9\" (UID: \"58309a66-dfdf-4793-bbc5-5c7b33ca111e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9" Nov 03 21:32:12.764195 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.764143 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9" Nov 03 21:32:12.970302 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:12.970264 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq"] Nov 03 21:32:12.975657 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:32:12.975626 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9204dab2_ccbe_4405_b9c7_666f23309bbd.slice/crio-9b7b77f5cd7d973fb88007c827a2c4a9f687c103fde18b61e41d3216e7593fea WatchSource:0}: Error finding container 9b7b77f5cd7d973fb88007c827a2c4a9f687c103fde18b61e41d3216e7593fea: Status 404 returned error can't find the container with id 9b7b77f5cd7d973fb88007c827a2c4a9f687c103fde18b61e41d3216e7593fea Nov 03 21:32:13.112093 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.112005 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n"] Nov 03 21:32:13.115973 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.115954 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.147114 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.147081 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9"] Nov 03 21:32:13.152565 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:32:13.152534 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58309a66_dfdf_4793_bbc5_5c7b33ca111e.slice/crio-9ec0a8492615b1318cf3e1919daf7076f5cc09b4986ca8d58520027cc3a1b77f WatchSource:0}: Error finding container 9ec0a8492615b1318cf3e1919daf7076f5cc09b4986ca8d58520027cc3a1b77f: Status 404 returned error can't find the container with id 9ec0a8492615b1318cf3e1919daf7076f5cc09b4986ca8d58520027cc3a1b77f Nov 03 21:32:13.198054 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.195804 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n"] Nov 03 21:32:13.261137 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.261103 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a41ad0-openshift-default-dockercfg-dhkbv\"" Nov 03 21:32:13.262597 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.262574 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.262675 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.262604 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.262675 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.262622 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb35897b-74a4-4d21-99d7-20fbe57e4747-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.262830 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.262793 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsdbl\" (UniqueName: \"kubernetes.io/projected/bb35897b-74a4-4d21-99d7-20fbe57e4747-kube-api-access-jsdbl\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.262997 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.262883 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.263078 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.262986 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.263078 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.263042 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.263190 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.263082 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.263258 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.263240 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.364336 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.364249 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.364519 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.364342 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.364519 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.364372 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.364519 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.364407 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb35897b-74a4-4d21-99d7-20fbe57e4747-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.364519 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.364468 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsdbl\" (UniqueName: \"kubernetes.io/projected/bb35897b-74a4-4d21-99d7-20fbe57e4747-kube-api-access-jsdbl\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.364519 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.364507 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.364813 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.364573 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.364813 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.364602 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.364813 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.364638 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.364813 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.364771 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.365072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.364809 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.365142 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.365119 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.365181 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.365145 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.365379 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.365300 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb35897b-74a4-4d21-99d7-20fbe57e4747-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.367250 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.367083 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.368785 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.368765 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.382352 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.382331 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:13.425977 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.423806 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsdbl\" (UniqueName: \"kubernetes.io/projected/bb35897b-74a4-4d21-99d7-20fbe57e4747-kube-api-access-jsdbl\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.429427 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.429395 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:13.471223 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.471185 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9" event={"ID":"58309a66-dfdf-4793-bbc5-5c7b33ca111e","Type":"ContainerStarted","Data":"9f5a1f12e08d4b55e677b1cf3391d4a38f0e438d8de8f3a51ad43bb4a0b7e449"} Nov 03 21:32:13.471432 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.471251 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9" event={"ID":"58309a66-dfdf-4793-bbc5-5c7b33ca111e","Type":"ContainerStarted","Data":"9ec0a8492615b1318cf3e1919daf7076f5cc09b4986ca8d58520027cc3a1b77f"} Nov 03 21:32:13.471432 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.471279 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9" Nov 03 21:32:13.473260 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.473229 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq" event={"ID":"9204dab2-ccbe-4405-b9c7-666f23309bbd","Type":"ContainerStarted","Data":"5b101cf0b10f79b1cb8d3d7fe1f774e56a2988ca2e1bad943c2632c3c8ba683b"} Nov 03 21:32:13.473391 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.473285 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq" event={"ID":"9204dab2-ccbe-4405-b9c7-666f23309bbd","Type":"ContainerStarted","Data":"9b7b77f5cd7d973fb88007c827a2c4a9f687c103fde18b61e41d3216e7593fea"} Nov 03 21:32:13.473391 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.473312 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq" Nov 03 21:32:13.569669 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.569636 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:13.581838 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.581775 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9" podStartSLOduration=1.581755945 podStartE2EDuration="1.581755945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:13.581357638 +0000 UTC m=+2682.741429909" watchObservedRunningTime="2026-11-03 21:32:13.581755945 +0000 UTC m=+2682.741828218" Nov 03 21:32:13.693506 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.693440 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq" podStartSLOduration=1.693417449 podStartE2EDuration="1.693417449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:13.691745002 +0000 UTC m=+2682.851817282" watchObservedRunningTime="2026-11-03 21:32:13.693417449 +0000 UTC m=+2682.853489722" Nov 03 21:32:13.729368 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:13.729326 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:14.213775 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:14.213726 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n"] Nov 03 21:32:14.255642 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:32:14.255606 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb35897b_74a4_4d21_99d7_20fbe57e4747.slice/crio-ecd6be01f76169b7bc3c3cde2ab1ff4d72d360067c7bfb37ccff721bf475f8e4 WatchSource:0}: Error finding container ecd6be01f76169b7bc3c3cde2ab1ff4d72d360067c7bfb37ccff721bf475f8e4: Status 404 returned error can't find the container with id ecd6be01f76169b7bc3c3cde2ab1ff4d72d360067c7bfb37ccff721bf475f8e4 Nov 03 21:32:14.258417 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:14.258385 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:32:14.258493 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:14.258454 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:32:14.258493 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:14.258482 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:32:14.490224 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:14.490091 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" event={"ID":"bb35897b-74a4-4d21-99d7-20fbe57e4747","Type":"ContainerStarted","Data":"d41780cf0da4d4d6058ab32c3cfd9cfb840955ce31be338529c128f1bbf5f93e"} Nov 03 21:32:14.490224 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:14.490170 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" event={"ID":"bb35897b-74a4-4d21-99d7-20fbe57e4747","Type":"ContainerStarted","Data":"ecd6be01f76169b7bc3c3cde2ab1ff4d72d360067c7bfb37ccff721bf475f8e4"} Nov 03 21:32:14.730072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:14.730032 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:15.731558 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:15.731523 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:32:15.732069 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:15.731578 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" podUID="bb35897b-74a4-4d21-99d7-20fbe57e4747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 21:32:15.875620 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:15.875585 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:16.730328 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:16.730286 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:32:16.730535 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:16.730356 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" podUID="bb35897b-74a4-4d21-99d7-20fbe57e4747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 21:32:17.729735 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:17.729696 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:32:17.730148 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:17.729751 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" podUID="bb35897b-74a4-4d21-99d7-20fbe57e4747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 21:32:18.335310 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:18.335276 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:18.730586 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:18.730549 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:32:18.730998 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:18.730605 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" podUID="bb35897b-74a4-4d21-99d7-20fbe57e4747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 21:32:19.730239 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:19.730200 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:32:19.730423 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:19.730257 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" podUID="bb35897b-74a4-4d21-99d7-20fbe57e4747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 21:32:20.713751 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:20.713715 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:20.730125 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:20.730076 4963 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:32:20.730341 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:20.730146 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" podUID="bb35897b-74a4-4d21-99d7-20fbe57e4747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 21:32:22.002669 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:22.002628 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:22.005159 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:22.003348 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:22.005159 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:22.004054 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:32:22.166325 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:22.166275 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" podStartSLOduration=9.166261022 podStartE2EDuration="9.166261022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:14.612234033 +0000 UTC m=+2683.772306305" watchObservedRunningTime="2026-11-03 21:32:22.166261022 +0000 UTC m=+2691.326333294" Nov 03 21:32:24.492345 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:24.492310 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9" Nov 03 21:32:24.493068 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:24.493048 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq" Nov 03 21:32:25.106432 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:25.106394 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:25.336787 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:25.336752 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:29.727325 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:29.727292 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:31.319940 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:31.319884 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:32:31.333257 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:31.333234 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:32:34.211468 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:34.211417 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:36.196061 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:36.196026 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:36.390750 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:36.390715 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:38.138744 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:38.138699 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:39.670588 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:39.670557 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:39.864548 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:39.864441 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:40.048043 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:40.047982 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:41.682615 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:41.682578 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:41.869649 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:41.869614 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:42.528544 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:32:42.528507 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:32:43.867917 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:43.867869 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:44.400443 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:44.400406 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:44.592766 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:44.592731 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:49.214473 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:49.214438 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:49.400823 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:49.400790 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:49.600775 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:49.600696 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:50.295395 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:50.295363 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:53.879301 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:53.879266 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:56.501684 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:56.501644 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:57.442848 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:57.442813 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:57.630576 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:57.630540 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:57.956847 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:57.956804 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:32:59.750284 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:32:59.750242 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:02.310734 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:02.310685 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:04.335885 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:04.335842 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:04.528848 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:04.528807 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:04.722564 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:04.722523 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:08.253722 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:08.253692 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:08.440455 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:08.440424 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:09.143154 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:09.143123 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:10.186263 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:10.186166 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:10.797208 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:10.797173 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:12.528441 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:33:12.528396 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:33:14.209661 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:14.209625 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:18.825418 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:18.825375 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:19.408064 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:19.408032 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:20.072971 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:20.070582 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:20.335657 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:20.335569 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:23.413656 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:23.413616 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:25.040624 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:25.040585 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:25.232320 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:25.232276 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:27.091126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:27.091035 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:28.305650 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:28.305610 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:28.499945 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:28.499883 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:28.717197 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:28.717164 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:28.848406 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:28.848366 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm"] Nov 03 21:33:28.892163 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:28.892127 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm"] Nov 03 21:33:28.892564 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:28.892490 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm" Nov 03 21:33:28.944820 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:28.944783 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-b22936\"" Nov 03 21:33:29.024402 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:29.024295 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c98j4\" (UniqueName: \"kubernetes.io/projected/422f25ab-7674-472a-bf7b-69f82b889a28-kube-api-access-c98j4\") pod \"payload-processing-e2e-explicit-b22936-9948895cd-c7csm\" (UID: \"422f25ab-7674-472a-bf7b-69f82b889a28\") " pod="openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm" Nov 03 21:33:29.024587 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:29.024471 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422f25ab-7674-472a-bf7b-69f82b889a28-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b22936-9948895cd-c7csm\" (UID: \"422f25ab-7674-472a-bf7b-69f82b889a28\") " pod="openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm" Nov 03 21:33:29.050499 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:29.050433 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-b22936-dockercfg-kc4nq\"" Nov 03 21:33:29.126209 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:29.126162 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422f25ab-7674-472a-bf7b-69f82b889a28-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b22936-9948895cd-c7csm\" (UID: \"422f25ab-7674-472a-bf7b-69f82b889a28\") " pod="openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm" Nov 03 21:33:29.126428 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:29.126303 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c98j4\" (UniqueName: \"kubernetes.io/projected/422f25ab-7674-472a-bf7b-69f82b889a28-kube-api-access-c98j4\") pod \"payload-processing-e2e-explicit-b22936-9948895cd-c7csm\" (UID: \"422f25ab-7674-472a-bf7b-69f82b889a28\") " pod="openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm" Nov 03 21:33:29.126937 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:29.126894 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422f25ab-7674-472a-bf7b-69f82b889a28-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b22936-9948895cd-c7csm\" (UID: \"422f25ab-7674-472a-bf7b-69f82b889a28\") " pod="openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm" Nov 03 21:33:29.184787 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:29.184755 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c98j4\" (UniqueName: \"kubernetes.io/projected/422f25ab-7674-472a-bf7b-69f82b889a28-kube-api-access-c98j4\") pod \"payload-processing-e2e-explicit-b22936-9948895cd-c7csm\" (UID: \"422f25ab-7674-472a-bf7b-69f82b889a28\") " pod="openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm" Nov 03 21:33:29.216577 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:29.216537 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm" Nov 03 21:33:29.602082 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:29.601988 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm"] Nov 03 21:33:29.606761 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:33:29.606726 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422f25ab_7674_472a_bf7b_69f82b889a28.slice/crio-9017b1cadd3383413e1e6183fcd9f40e04ebb54c8f4b74a7bd615d31d963fa1a WatchSource:0}: Error finding container 9017b1cadd3383413e1e6183fcd9f40e04ebb54c8f4b74a7bd615d31d963fa1a: Status 404 returned error can't find the container with id 9017b1cadd3383413e1e6183fcd9f40e04ebb54c8f4b74a7bd615d31d963fa1a Nov 03 21:33:30.008342 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:30.008295 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm" event={"ID":"422f25ab-7674-472a-bf7b-69f82b889a28","Type":"ContainerStarted","Data":"790dbe7bbc670d4fb47b3161ef09f456a1b0105b0f2ff00181438233e2b01998"} Nov 03 21:33:30.008522 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:30.008368 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm" event={"ID":"422f25ab-7674-472a-bf7b-69f82b889a28","Type":"ContainerStarted","Data":"9017b1cadd3383413e1e6183fcd9f40e04ebb54c8f4b74a7bd615d31d963fa1a"} Nov 03 21:33:30.008522 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:30.008496 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm" Nov 03 21:33:30.122345 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:30.122288 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm" podStartSLOduration=2.122272556 podStartE2EDuration="2.122272556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:33:30.118563984 +0000 UTC m=+2759.278636255" watchObservedRunningTime="2026-11-03 21:33:30.122272556 +0000 UTC m=+2759.282344827" Nov 03 21:33:33.307782 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:33.307746 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:36.547445 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:36.547408 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:37.721934 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.721890 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv"] Nov 03 21:33:37.726366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.726343 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:37.824938 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.824862 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-c70fca\"/\"openshift-service-ca.crt\"" Nov 03 21:33:37.824938 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.824879 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-c70fca\"/\"kube-root-ca.crt\"" Nov 03 21:33:37.824938 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.824877 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-c70fca\"/\"default-dockercfg-npcrx\"" Nov 03 21:33:37.825244 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.825010 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-c70fca\"/\"auth-test-model-c70fca-kserve-self-signed-certs\"" Nov 03 21:33:37.827484 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.827460 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv"] Nov 03 21:33:37.827574 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.827541 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca06cf22-420e-4bde-924a-13c7835a6593-tls-certs\") pod \"auth-test-model-c70fca-kserve-56455bd765-pqpkv\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:37.827613 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.827582 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-dshm\") pod \"auth-test-model-c70fca-kserve-56455bd765-pqpkv\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:37.827613 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.827598 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-tmp-dir\") pod \"auth-test-model-c70fca-kserve-56455bd765-pqpkv\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:37.827716 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.827699 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-model-cache\") pod \"auth-test-model-c70fca-kserve-56455bd765-pqpkv\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:37.827751 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.827729 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vld56\" (UniqueName: \"kubernetes.io/projected/ca06cf22-420e-4bde-924a-13c7835a6593-kube-api-access-vld56\") pod \"auth-test-model-c70fca-kserve-56455bd765-pqpkv\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:37.827879 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.827860 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-home\") pod \"auth-test-model-c70fca-kserve-56455bd765-pqpkv\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:37.862624 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.862590 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:37.928669 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.928632 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-home\") pod \"auth-test-model-c70fca-kserve-56455bd765-pqpkv\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:37.928883 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.928752 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca06cf22-420e-4bde-924a-13c7835a6593-tls-certs\") pod \"auth-test-model-c70fca-kserve-56455bd765-pqpkv\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:37.928883 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.928784 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-dshm\") pod \"auth-test-model-c70fca-kserve-56455bd765-pqpkv\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:37.928883 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.928799 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-tmp-dir\") pod \"auth-test-model-c70fca-kserve-56455bd765-pqpkv\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:37.929129 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.928936 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-model-cache\") pod \"auth-test-model-c70fca-kserve-56455bd765-pqpkv\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:37.929129 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.928990 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vld56\" (UniqueName: \"kubernetes.io/projected/ca06cf22-420e-4bde-924a-13c7835a6593-kube-api-access-vld56\") pod \"auth-test-model-c70fca-kserve-56455bd765-pqpkv\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:37.929352 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.929326 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-model-cache\") pod \"auth-test-model-c70fca-kserve-56455bd765-pqpkv\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:37.929586 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.929563 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-home\") pod \"auth-test-model-c70fca-kserve-56455bd765-pqpkv\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:37.929734 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.929707 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-tmp-dir\") pod \"auth-test-model-c70fca-kserve-56455bd765-pqpkv\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:37.931413 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.931390 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-dshm\") pod \"auth-test-model-c70fca-kserve-56455bd765-pqpkv\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:37.931838 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.931821 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca06cf22-420e-4bde-924a-13c7835a6593-tls-certs\") pod \"auth-test-model-c70fca-kserve-56455bd765-pqpkv\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:37.993224 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:37.993154 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vld56\" (UniqueName: \"kubernetes.io/projected/ca06cf22-420e-4bde-924a-13c7835a6593-kube-api-access-vld56\") pod \"auth-test-model-c70fca-kserve-56455bd765-pqpkv\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:38.038584 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:38.038535 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:38.049817 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:38.049780 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:38.442194 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:38.442156 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv"] Nov 03 21:33:38.447585 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:33:38.447558 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca06cf22_420e_4bde_924a_13c7835a6593.slice/crio-e7e01ab8526431bc7a7de35f192336751e31b5debf62187c8567ba47f4e81232 WatchSource:0}: Error finding container e7e01ab8526431bc7a7de35f192336751e31b5debf62187c8567ba47f4e81232: Status 404 returned error can't find the container with id e7e01ab8526431bc7a7de35f192336751e31b5debf62187c8567ba47f4e81232 Nov 03 21:33:39.070572 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:39.070535 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" event={"ID":"ca06cf22-420e-4bde-924a-13c7835a6593","Type":"ContainerStarted","Data":"f69b6b3a23f74bc6ec96acaed26acf1e98723f38f11ef7fdc751482b5ba0e678"} Nov 03 21:33:39.070572 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:39.070580 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" event={"ID":"ca06cf22-420e-4bde-924a-13c7835a6593","Type":"ContainerStarted","Data":"e7e01ab8526431bc7a7de35f192336751e31b5debf62187c8567ba47f4e81232"} Nov 03 21:33:39.218626 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:39.218568 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" podStartSLOduration=2.218553956 podStartE2EDuration="2.218553956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:33:39.2183718 +0000 UTC m=+2768.378444071" watchObservedRunningTime="2026-11-03 21:33:39.218553956 +0000 UTC m=+2768.378626227" Nov 03 21:33:40.039603 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:40.039567 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:41.018246 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:41.018217 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm" Nov 03 21:33:42.513592 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:42.513557 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:42.529564 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:33:42.529518 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:33:47.103844 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:47.103786 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:48.039009 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:48.038979 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:48.051788 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:48.051754 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:48.140765 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:48.140730 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:33:52.012490 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:52.012461 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:56.737622 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:56.737575 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:33:59.651440 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:33:59.651409 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:00.900740 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:00.900606 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:01.280004 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:01.279970 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:01.554729 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:01.554617 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:05.016509 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:05.016472 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:05.237072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:05.237038 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:06.249878 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:06.249835 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:07.045842 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:07.045807 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:08.223444 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:08.223407 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:08.411513 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:08.411483 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:11.076114 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:11.076072 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:12.529783 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:34:12.529741 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:34:15.713103 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:15.713063 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:20.202993 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:20.202962 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:24.448761 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:24.448721 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf"] Nov 03 21:34:24.454368 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:24.454340 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:34:24.493811 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:24.493773 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c5e674-acbe-4286-9ea0-8a326eb831b1-openshift-service-ca\") pod \"maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf\" (UID: \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:34:24.494016 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:24.493884 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7c5e674-acbe-4286-9ea0-8a326eb831b1-maas-api-tls\") pod \"maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf\" (UID: \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:34:24.494016 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:24.493992 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8vls\" (UniqueName: \"kubernetes.io/projected/e7c5e674-acbe-4286-9ea0-8a326eb831b1-kube-api-access-c8vls\") pod \"maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf\" (UID: \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:34:24.494115 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:24.494052 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7c5e674-acbe-4286-9ea0-8a326eb831b1-metrics-certs\") pod \"maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf\" (UID: \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:34:24.504348 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:24.504304 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-b22936-serving-cert\"" Nov 03 21:34:24.556496 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:24.556455 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf"] Nov 03 21:34:24.595793 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:24.595725 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c5e674-acbe-4286-9ea0-8a326eb831b1-openshift-service-ca\") pod \"maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf\" (UID: \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:34:24.596099 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:24.595864 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7c5e674-acbe-4286-9ea0-8a326eb831b1-maas-api-tls\") pod \"maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf\" (UID: \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:34:24.596184 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:34:24.596119 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-serving-cert: secret "maas-api-e2e-explicit-b22936-serving-cert" not found Nov 03 21:34:24.596246 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:34:24.596217 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7c5e674-acbe-4286-9ea0-8a326eb831b1-maas-api-tls podName:e7c5e674-acbe-4286-9ea0-8a326eb831b1 nodeName:}" failed. No retries permitted until 2026-11-03 21:34:25.096191968 +0000 UTC m=+2814.256264241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e7c5e674-acbe-4286-9ea0-8a326eb831b1-maas-api-tls") pod "maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" (UID: "e7c5e674-acbe-4286-9ea0-8a326eb831b1") : secret "maas-api-e2e-explicit-b22936-serving-cert" not found Nov 03 21:34:24.596391 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:24.596121 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8vls\" (UniqueName: \"kubernetes.io/projected/e7c5e674-acbe-4286-9ea0-8a326eb831b1-kube-api-access-c8vls\") pod \"maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf\" (UID: \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:34:24.596523 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:24.596502 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7c5e674-acbe-4286-9ea0-8a326eb831b1-metrics-certs\") pod \"maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf\" (UID: \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:34:24.597585 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:24.597544 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c5e674-acbe-4286-9ea0-8a326eb831b1-openshift-service-ca\") pod \"maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf\" (UID: \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:34:24.600447 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:24.600423 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7c5e674-acbe-4286-9ea0-8a326eb831b1-metrics-certs\") pod \"maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf\" (UID: \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:34:24.656593 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:24.656559 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8vls\" (UniqueName: \"kubernetes.io/projected/e7c5e674-acbe-4286-9ea0-8a326eb831b1-kube-api-access-c8vls\") pod \"maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf\" (UID: \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:34:24.677302 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:24.677265 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:24.863723 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:24.863635 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:25.046524 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:25.046488 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:25.102842 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:25.102799 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7c5e674-acbe-4286-9ea0-8a326eb831b1-maas-api-tls\") pod \"maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf\" (UID: \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:34:25.105357 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:25.105323 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7c5e674-acbe-4286-9ea0-8a326eb831b1-maas-api-tls\") pod \"maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf\" (UID: \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:34:25.369160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:25.369121 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:34:25.754548 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:25.754501 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf"] Nov 03 21:34:25.760018 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:34:25.759990 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c5e674_acbe_4286_9ea0_8a326eb831b1.slice/crio-2b16da4edb2ff4eb8ab128e5a0f2846344694f8f65ef5fba7174616120fc6568 WatchSource:0}: Error finding container 2b16da4edb2ff4eb8ab128e5a0f2846344694f8f65ef5fba7174616120fc6568: Status 404 returned error can't find the container with id 2b16da4edb2ff4eb8ab128e5a0f2846344694f8f65ef5fba7174616120fc6568 Nov 03 21:34:26.385964 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:26.385900 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" event={"ID":"e7c5e674-acbe-4286-9ea0-8a326eb831b1","Type":"ContainerStarted","Data":"2b16da4edb2ff4eb8ab128e5a0f2846344694f8f65ef5fba7174616120fc6568"} Nov 03 21:34:27.394091 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:27.394052 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" event={"ID":"e7c5e674-acbe-4286-9ea0-8a326eb831b1","Type":"ContainerStarted","Data":"021b4764a325f96daef5ddc280c047fbded9af8a06bfca12ca59904c00d0efc0"} Nov 03 21:34:27.394780 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:27.394121 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:34:27.507854 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:27.507793 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" podStartSLOduration=2.619129536 podStartE2EDuration="3.507776818s" totalImagesPullingTime="888.647282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:27.506005459 +0000 UTC m=+2816.666077727" watchObservedRunningTime="2026-11-03 21:34:27.507776818 +0000 UTC m=+2816.667849089" Nov 03 21:34:29.379082 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:29.379045 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:29.571589 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:29.571553 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:30.771105 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:30.771072 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:30.959363 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:30.959319 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:33.410144 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:33.410112 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:34:34.030088 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:34.030044 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:36.891337 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:36.891303 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:38.517553 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:38.517520 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:42.529395 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:34:42.529311 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:34:43.100849 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:43.100813 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:43.578916 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:43.578877 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:47.580721 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:47.580688 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:52.129233 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:52.129195 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:56.717769 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:56.717728 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:34:58.667251 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:34:58.667221 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:01.203763 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:01.203727 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:05.719048 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:05.719008 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:09.873054 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:09.873008 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z"] Nov 03 21:35:09.877709 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:09.877683 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 21:35:09.968181 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:09.968144 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z\" (UID: \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 21:35:09.968412 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:09.968220 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z\" (UID: \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 21:35:09.968412 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:09.968344 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jwz8\" (UniqueName: \"kubernetes.io/projected/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-kube-api-access-7jwz8\") pod \"maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z\" (UID: \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 21:35:09.968573 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:09.968462 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z\" (UID: \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 21:35:09.975753 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:09.975722 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-714979-serving-cert\"" Nov 03 21:35:09.979845 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:09.979816 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z"] Nov 03 21:35:10.097798 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:10.097762 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z\" (UID: \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 21:35:10.098009 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:10.097865 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z\" (UID: \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 21:35:10.100962 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:10.098102 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jwz8\" (UniqueName: \"kubernetes.io/projected/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-kube-api-access-7jwz8\") pod \"maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z\" (UID: \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 21:35:10.100962 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:35:10.098213 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-serving-cert: secret "maas-api-e2e-shared-b-w1-714979-serving-cert" not found Nov 03 21:35:10.100962 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:35:10.098304 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-maas-api-tls podName:0705edee-7b08-4a0d-b47c-7b8ce7f3279a nodeName:}" failed. No retries permitted until 2026-11-03 21:35:10.598284124 +0000 UTC m=+2859.758356377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-maas-api-tls") pod "maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" (UID: "0705edee-7b08-4a0d-b47c-7b8ce7f3279a") : secret "maas-api-e2e-shared-b-w1-714979-serving-cert" not found Nov 03 21:35:10.100962 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:10.098327 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z\" (UID: \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 21:35:10.101374 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:10.101299 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z\" (UID: \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 21:35:10.106698 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:10.106658 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:10.118311 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:10.118187 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z\" (UID: \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 21:35:10.160985 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:10.160893 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jwz8\" (UniqueName: \"kubernetes.io/projected/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-kube-api-access-7jwz8\") pod \"maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z\" (UID: \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 21:35:10.291037 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:10.290988 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:10.481209 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:10.481174 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:10.605658 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:10.605619 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z\" (UID: \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 21:35:10.608467 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:10.608432 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z\" (UID: \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 21:35:10.793915 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:10.793808 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 21:35:11.182349 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:11.182252 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z"] Nov 03 21:35:11.188543 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:35:11.188514 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0705edee_7b08_4a0d_b47c_7b8ce7f3279a.slice/crio-93e1fb3ba409e46b2298aa8930c9a6103225809a8b85bec2a2bb08d124bbb9c9 WatchSource:0}: Error finding container 93e1fb3ba409e46b2298aa8930c9a6103225809a8b85bec2a2bb08d124bbb9c9: Status 404 returned error can't find the container with id 93e1fb3ba409e46b2298aa8930c9a6103225809a8b85bec2a2bb08d124bbb9c9 Nov 03 21:35:11.711762 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:11.711725 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" event={"ID":"0705edee-7b08-4a0d-b47c-7b8ce7f3279a","Type":"ContainerStarted","Data":"93e1fb3ba409e46b2298aa8930c9a6103225809a8b85bec2a2bb08d124bbb9c9"} Nov 03 21:35:12.528666 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:35:12.528629 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:35:12.720963 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:12.720922 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" event={"ID":"0705edee-7b08-4a0d-b47c-7b8ce7f3279a","Type":"ContainerStarted","Data":"0726097228a71e7cc1704fa2f0c4b6a99773034a443e976df0453921b0c032a7"} Nov 03 21:35:12.721144 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:12.720991 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 21:35:12.835723 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:12.835619 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" podStartSLOduration=2.8727029010000003 podStartE2EDuration="3.835603369s" totalImagesPullingTime="962.900468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:12.83285761 +0000 UTC m=+2861.992929882" watchObservedRunningTime="2026-11-03 21:35:12.835603369 +0000 UTC m=+2861.995675640" Nov 03 21:35:13.017820 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:13.017786 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb"] Nov 03 21:35:13.018047 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:13.018023 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" podUID="81c8de6a-8867-40ba-b88c-cb0d73475112" containerName="istio-proxy" containerID="cri-o://60f60e8db8dbca66114b5976a30eef8f6507c890fa9408fdf8c8d41fd21444d7" gracePeriod=30 Nov 03 21:35:14.809140 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:14.809107 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:18.293424 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.293357 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 21:35:18.409860 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.409827 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81c8de6a-8867-40ba-b88c-cb0d73475112-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81c8de6a-8867-40ba-b88c-cb0d73475112-istiod-ca-cert\") pod \"81c8de6a-8867-40ba-b88c-cb0d73475112\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " Nov 03 21:35:18.410072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.409887 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-data\") pod \"81c8de6a-8867-40ba-b88c-cb0d73475112\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " Nov 03 21:35:18.410072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.409954 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81c8de6a-8867-40ba-b88c-cb0d73475112-kube-api-access-dsjpx\" (UniqueName: \"kubernetes.io/projected/81c8de6a-8867-40ba-b88c-cb0d73475112-kube-api-access-dsjpx\") pod \"81c8de6a-8867-40ba-b88c-cb0d73475112\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " Nov 03 21:35:18.410072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.409991 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-workload-certs\") pod \"81c8de6a-8867-40ba-b88c-cb0d73475112\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " Nov 03 21:35:18.410072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.410026 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-credential-socket\") pod \"81c8de6a-8867-40ba-b88c-cb0d73475112\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " Nov 03 21:35:18.410072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.410059 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-token\" (UniqueName: \"kubernetes.io/projected/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-token\") pod \"81c8de6a-8867-40ba-b88c-cb0d73475112\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " Nov 03 21:35:18.410354 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.410209 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-podinfo\") pod \"81c8de6a-8867-40ba-b88c-cb0d73475112\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " Nov 03 21:35:18.410354 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.410232 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-data" pod "81c8de6a-8867-40ba-b88c-cb0d73475112" (UID: "81c8de6a-8867-40ba-b88c-cb0d73475112"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:18.410354 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.410257 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-workload-certs" pod "81c8de6a-8867-40ba-b88c-cb0d73475112" (UID: "81c8de6a-8867-40ba-b88c-cb0d73475112"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:18.410354 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.410293 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-credential-socket" pod "81c8de6a-8867-40ba-b88c-cb0d73475112" (UID: "81c8de6a-8867-40ba-b88c-cb0d73475112"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:18.410354 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.410277 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-workload-socket\") pod \"81c8de6a-8867-40ba-b88c-cb0d73475112\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " Nov 03 21:35:18.410354 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.410325 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81c8de6a-8867-40ba-b88c-cb0d73475112-istiod-ca-cert" pod "81c8de6a-8867-40ba-b88c-cb0d73475112" (UID: "81c8de6a-8867-40ba-b88c-cb0d73475112"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:35:18.410612 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.410394 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-envoy\") pod \"81c8de6a-8867-40ba-b88c-cb0d73475112\" (UID: \"81c8de6a-8867-40ba-b88c-cb0d73475112\") " Nov 03 21:35:18.410612 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.410493 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-workload-socket" pod "81c8de6a-8867-40ba-b88c-cb0d73475112" (UID: "81c8de6a-8867-40ba-b88c-cb0d73475112"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:18.411014 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.410990 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:18.411128 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.411017 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:18.411128 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.411035 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:18.411128 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.411050 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:18.411128 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.411064 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81c8de6a-8867-40ba-b88c-cb0d73475112-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:18.412391 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.412364 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-podinfo" pod "81c8de6a-8867-40ba-b88c-cb0d73475112" (UID: "81c8de6a-8867-40ba-b88c-cb0d73475112"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:35:18.412496 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.412435 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c8de6a-8867-40ba-b88c-cb0d73475112-kube-api-access-dsjpx" pod "81c8de6a-8867-40ba-b88c-cb0d73475112" (UID: "81c8de6a-8867-40ba-b88c-cb0d73475112"). InnerVolumeSpecName "kube-api-access-dsjpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:35:18.412496 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.412458 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-token" pod "81c8de6a-8867-40ba-b88c-cb0d73475112" (UID: "81c8de6a-8867-40ba-b88c-cb0d73475112"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:35:18.412496 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.412476 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-envoy" pod "81c8de6a-8867-40ba-b88c-cb0d73475112" (UID: "81c8de6a-8867-40ba-b88c-cb0d73475112"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:18.512502 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.512462 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:18.512502 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.512494 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:18.512502 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.512504 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsjpx\" (UniqueName: \"kubernetes.io/projected/81c8de6a-8867-40ba-b88c-cb0d73475112-kube-api-access-dsjpx\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:18.512746 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.512514 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81c8de6a-8867-40ba-b88c-cb0d73475112-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:18.732323 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.732287 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 21:35:18.758748 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.758710 4963 generic.go:356] "Generic (PLEG): container finished" podID="81c8de6a-8867-40ba-b88c-cb0d73475112" containerID="60f60e8db8dbca66114b5976a30eef8f6507c890fa9408fdf8c8d41fd21444d7" exitCode=0 Nov 03 21:35:18.758973 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.758763 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" Nov 03 21:35:18.758973 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.758795 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" event={"ID":"81c8de6a-8867-40ba-b88c-cb0d73475112","Type":"ContainerDied","Data":"60f60e8db8dbca66114b5976a30eef8f6507c890fa9408fdf8c8d41fd21444d7"} Nov 03 21:35:18.758973 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.758868 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb" event={"ID":"81c8de6a-8867-40ba-b88c-cb0d73475112","Type":"ContainerDied","Data":"b2327e5256359edc7cafb0f94361e3a4f4eca32985e26c24bff298f62a6877cb"} Nov 03 21:35:18.758973 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.758898 4963 scope.go:122] "RemoveContainer" containerID="60f60e8db8dbca66114b5976a30eef8f6507c890fa9408fdf8c8d41fd21444d7" Nov 03 21:35:18.770722 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.770700 4963 scope.go:122] "RemoveContainer" containerID="60f60e8db8dbca66114b5976a30eef8f6507c890fa9408fdf8c8d41fd21444d7" Nov 03 21:35:18.771079 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:35:18.771052 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f60e8db8dbca66114b5976a30eef8f6507c890fa9408fdf8c8d41fd21444d7\": container with ID starting with 60f60e8db8dbca66114b5976a30eef8f6507c890fa9408fdf8c8d41fd21444d7 not found: ID does not exist" containerID="60f60e8db8dbca66114b5976a30eef8f6507c890fa9408fdf8c8d41fd21444d7" Nov 03 21:35:18.771178 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.771089 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f60e8db8dbca66114b5976a30eef8f6507c890fa9408fdf8c8d41fd21444d7"} err="failed to get container status \"60f60e8db8dbca66114b5976a30eef8f6507c890fa9408fdf8c8d41fd21444d7\": rpc error: code = NotFound desc = could not find container \"60f60e8db8dbca66114b5976a30eef8f6507c890fa9408fdf8c8d41fd21444d7\": container with ID starting with 60f60e8db8dbca66114b5976a30eef8f6507c890fa9408fdf8c8d41fd21444d7 not found: ID does not exist" Nov 03 21:35:18.954953 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.954888 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb"] Nov 03 21:35:18.958521 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.958494 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-rl6tb"] Nov 03 21:35:18.978729 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:18.978687 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81c8de6a-8867-40ba-b88c-cb0d73475112" path="/var/lib/kubelet/pods/81c8de6a-8867-40ba-b88c-cb0d73475112/volumes" Nov 03 21:35:19.361048 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:19.361006 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:23.858483 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:23.858439 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:27.297163 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.297125 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29"] Nov 03 21:35:27.297738 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.297721 4963 cpu_manager.go:414] "containerMap: removing container" podUID="81c8de6a-8867-40ba-b88c-cb0d73475112" containerName="istio-proxy" Nov 03 21:35:27.297799 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.297740 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81c8de6a-8867-40ba-b88c-cb0d73475112" containerName="istio-proxy" Nov 03 21:35:27.297871 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.297861 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="81c8de6a-8867-40ba-b88c-cb0d73475112" containerName="istio-proxy" Nov 03 21:35:27.303207 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.303187 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.361347 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.361312 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29"] Nov 03 21:35:27.405519 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.405487 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-a1c634\"/\"openshift-service-ca.crt\"" Nov 03 21:35:27.405700 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.405547 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-a1c634\"/\"kube-root-ca.crt\"" Nov 03 21:35:27.405700 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.405555 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-a1c634\"/\"default-dockercfg-zhztf\"" Nov 03 21:35:27.405783 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.405739 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-a1c634\"/\"auth-test-model-a1c634-kserve-self-signed-certs\"" Nov 03 21:35:27.419623 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.419590 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-model-cache\") pod \"auth-test-model-a1c634-kserve-5d6469bdb5-nds29\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.419730 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.419632 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-home\") pod \"auth-test-model-a1c634-kserve-5d6469bdb5-nds29\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.419797 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.419745 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-tmp-dir\") pod \"auth-test-model-a1c634-kserve-5d6469bdb5-nds29\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.419869 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.419855 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/276ee27d-7d04-4a02-94cd-2cac042b72a6-tls-certs\") pod \"auth-test-model-a1c634-kserve-5d6469bdb5-nds29\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.419949 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.419895 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr46c\" (UniqueName: \"kubernetes.io/projected/276ee27d-7d04-4a02-94cd-2cac042b72a6-kube-api-access-xr46c\") pod \"auth-test-model-a1c634-kserve-5d6469bdb5-nds29\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.419991 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.419965 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-dshm\") pod \"auth-test-model-a1c634-kserve-5d6469bdb5-nds29\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.443089 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.443052 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:27.521473 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.521427 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr46c\" (UniqueName: \"kubernetes.io/projected/276ee27d-7d04-4a02-94cd-2cac042b72a6-kube-api-access-xr46c\") pod \"auth-test-model-a1c634-kserve-5d6469bdb5-nds29\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.521692 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.521491 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-dshm\") pod \"auth-test-model-a1c634-kserve-5d6469bdb5-nds29\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.521692 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.521561 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-model-cache\") pod \"auth-test-model-a1c634-kserve-5d6469bdb5-nds29\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.521692 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.521590 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-home\") pod \"auth-test-model-a1c634-kserve-5d6469bdb5-nds29\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.521692 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.521666 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-tmp-dir\") pod \"auth-test-model-a1c634-kserve-5d6469bdb5-nds29\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.521920 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.521797 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/276ee27d-7d04-4a02-94cd-2cac042b72a6-tls-certs\") pod \"auth-test-model-a1c634-kserve-5d6469bdb5-nds29\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.522221 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.522175 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-home\") pod \"auth-test-model-a1c634-kserve-5d6469bdb5-nds29\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.522323 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.522299 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-model-cache\") pod \"auth-test-model-a1c634-kserve-5d6469bdb5-nds29\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.522476 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.522446 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-tmp-dir\") pod \"auth-test-model-a1c634-kserve-5d6469bdb5-nds29\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.524194 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.524172 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-dshm\") pod \"auth-test-model-a1c634-kserve-5d6469bdb5-nds29\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.524727 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.524708 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/276ee27d-7d04-4a02-94cd-2cac042b72a6-tls-certs\") pod \"auth-test-model-a1c634-kserve-5d6469bdb5-nds29\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.582957 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.582859 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr46c\" (UniqueName: \"kubernetes.io/projected/276ee27d-7d04-4a02-94cd-2cac042b72a6-kube-api-access-xr46c\") pod \"auth-test-model-a1c634-kserve-5d6469bdb5-nds29\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:27.617001 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:27.616959 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:28.008309 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:28.008209 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29"] Nov 03 21:35:28.013654 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:35:28.013621 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod276ee27d_7d04_4a02_94cd_2cac042b72a6.slice/crio-eb3aba7d5142ed34db44b8801d61b3c8a6a61526e8627ad56c3819cbef247ab5 WatchSource:0}: Error finding container eb3aba7d5142ed34db44b8801d61b3c8a6a61526e8627ad56c3819cbef247ab5: Status 404 returned error can't find the container with id eb3aba7d5142ed34db44b8801d61b3c8a6a61526e8627ad56c3819cbef247ab5 Nov 03 21:35:28.415140 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:28.415010 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:28.603555 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:28.603516 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:28.843543 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:28.843501 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" event={"ID":"276ee27d-7d04-4a02-94cd-2cac042b72a6","Type":"ContainerStarted","Data":"f78704e2ae5d67efcde02154f54d5040cb250f992e27bb075cf450accf5a888a"} Nov 03 21:35:28.843767 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:28.843559 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" event={"ID":"276ee27d-7d04-4a02-94cd-2cac042b72a6","Type":"ContainerStarted","Data":"eb3aba7d5142ed34db44b8801d61b3c8a6a61526e8627ad56c3819cbef247ab5"} Nov 03 21:35:28.957101 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:28.957048 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" podStartSLOduration=1.957033394 podStartE2EDuration="1.957033394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:28.955109022 +0000 UTC m=+2878.115181294" watchObservedRunningTime="2026-11-03 21:35:28.957033394 +0000 UTC m=+2878.117105704" Nov 03 21:35:29.618065 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:29.618026 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:32.998665 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:32.998622 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:37.215957 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:37.215915 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:37.401026 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:37.400960 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:37.588965 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:37.588850 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:37.618099 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:37.618051 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:37.631192 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:37.631164 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:37.915721 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:37.915644 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:35:42.295135 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:42.295104 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:42.527862 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:35:42.527824 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:35:46.823632 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:46.823588 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:51.314049 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:51.314008 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:55.797506 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:55.797448 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:35:56.956978 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:56.956944 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v"] Nov 03 21:35:56.957461 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:35:56.957156 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" podUID="1650f263-d416-4ba6-8067-63828b7c64d3" containerName="istio-proxy" containerID="cri-o://da3c6454c3d7a5efabccee7b9189ab13ca04082e0cb15ca672ea225b58a57e3d" gracePeriod=30 Nov 03 21:36:00.238473 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:00.238433 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx"] Nov 03 21:36:00.239051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:00.238650 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" podUID="7b117e23-def9-49c7-a6bd-db4ccb5c1a12" containerName="istio-proxy" containerID="cri-o://a1c35caeb52f410821f40603934300bc2258c6e5e2bd7089952d88dc52b930fd" gracePeriod=30 Nov 03 21:36:00.303912 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:00.303862 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:01.750060 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:01.750028 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:01.968000 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:01.967944 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:02.086621 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.086588 4963 generic.go:356] "Generic (PLEG): container finished" podID="1650f263-d416-4ba6-8067-63828b7c64d3" containerID="da3c6454c3d7a5efabccee7b9189ab13ca04082e0cb15ca672ea225b58a57e3d" exitCode=0 Nov 03 21:36:02.086784 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.086661 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" event={"ID":"1650f263-d416-4ba6-8067-63828b7c64d3","Type":"ContainerDied","Data":"da3c6454c3d7a5efabccee7b9189ab13ca04082e0cb15ca672ea225b58a57e3d"} Nov 03 21:36:02.243131 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.243100 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:36:02.362317 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.362234 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-workload-certs\") pod \"1650f263-d416-4ba6-8067-63828b7c64d3\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " Nov 03 21:36:02.362479 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.362318 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1650f263-d416-4ba6-8067-63828b7c64d3-istio-token\" (UniqueName: \"kubernetes.io/projected/1650f263-d416-4ba6-8067-63828b7c64d3-istio-token\") pod \"1650f263-d416-4ba6-8067-63828b7c64d3\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " Nov 03 21:36:02.362479 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.362377 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1650f263-d416-4ba6-8067-63828b7c64d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1650f263-d416-4ba6-8067-63828b7c64d3-istiod-ca-cert\") pod \"1650f263-d416-4ba6-8067-63828b7c64d3\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " Nov 03 21:36:02.362479 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.362407 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-workload-socket\") pod \"1650f263-d416-4ba6-8067-63828b7c64d3\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " Nov 03 21:36:02.362479 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.362425 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-istio-data\") pod \"1650f263-d416-4ba6-8067-63828b7c64d3\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " Nov 03 21:36:02.362698 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.362486 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-credential-socket\") pod \"1650f263-d416-4ba6-8067-63828b7c64d3\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " Nov 03 21:36:02.362698 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.362567 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-istio-envoy\") pod \"1650f263-d416-4ba6-8067-63828b7c64d3\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " Nov 03 21:36:02.362698 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.362574 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-workload-certs" pod "1650f263-d416-4ba6-8067-63828b7c64d3" (UID: "1650f263-d416-4ba6-8067-63828b7c64d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:02.362870 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.362793 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-workload-socket" pod "1650f263-d416-4ba6-8067-63828b7c64d3" (UID: "1650f263-d416-4ba6-8067-63828b7c64d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:02.362870 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.362809 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-credential-socket" pod "1650f263-d416-4ba6-8067-63828b7c64d3" (UID: "1650f263-d416-4ba6-8067-63828b7c64d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:02.362870 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.362784 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-istio-data" pod "1650f263-d416-4ba6-8067-63828b7c64d3" (UID: "1650f263-d416-4ba6-8067-63828b7c64d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:02.362870 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.362840 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1650f263-d416-4ba6-8067-63828b7c64d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1650f263-d416-4ba6-8067-63828b7c64d3-istio-podinfo\") pod \"1650f263-d416-4ba6-8067-63828b7c64d3\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " Nov 03 21:36:02.363113 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.362875 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1650f263-d416-4ba6-8067-63828b7c64d3-kube-api-access-2q29w\" (UniqueName: \"kubernetes.io/projected/1650f263-d416-4ba6-8067-63828b7c64d3-kube-api-access-2q29w\") pod \"1650f263-d416-4ba6-8067-63828b7c64d3\" (UID: \"1650f263-d416-4ba6-8067-63828b7c64d3\") " Nov 03 21:36:02.363113 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.362888 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1650f263-d416-4ba6-8067-63828b7c64d3-istiod-ca-cert" pod "1650f263-d416-4ba6-8067-63828b7c64d3" (UID: "1650f263-d416-4ba6-8067-63828b7c64d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:36:02.363410 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.363387 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:02.363583 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.363412 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1650f263-d416-4ba6-8067-63828b7c64d3-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:02.363583 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.363426 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:02.363583 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.363441 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:02.363583 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.363455 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:02.365083 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.365056 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-istio-envoy" pod "1650f263-d416-4ba6-8067-63828b7c64d3" (UID: "1650f263-d416-4ba6-8067-63828b7c64d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:02.365170 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.365141 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1650f263-d416-4ba6-8067-63828b7c64d3-istio-podinfo" pod "1650f263-d416-4ba6-8067-63828b7c64d3" (UID: "1650f263-d416-4ba6-8067-63828b7c64d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:36:02.365318 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.365301 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1650f263-d416-4ba6-8067-63828b7c64d3-kube-api-access-2q29w" pod "1650f263-d416-4ba6-8067-63828b7c64d3" (UID: "1650f263-d416-4ba6-8067-63828b7c64d3"). InnerVolumeSpecName "kube-api-access-2q29w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:36:02.365378 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.365326 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1650f263-d416-4ba6-8067-63828b7c64d3-istio-token" pod "1650f263-d416-4ba6-8067-63828b7c64d3" (UID: "1650f263-d416-4ba6-8067-63828b7c64d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:36:02.464498 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.464460 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1650f263-d416-4ba6-8067-63828b7c64d3-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:02.464498 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.464498 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1650f263-d416-4ba6-8067-63828b7c64d3-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:02.464697 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.464514 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q29w\" (UniqueName: \"kubernetes.io/projected/1650f263-d416-4ba6-8067-63828b7c64d3-kube-api-access-2q29w\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:02.464697 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:02.464530 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1650f263-d416-4ba6-8067-63828b7c64d3-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:03.094241 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:03.094206 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" Nov 03 21:36:03.094241 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:03.094234 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v" event={"ID":"1650f263-d416-4ba6-8067-63828b7c64d3","Type":"ContainerDied","Data":"17e1fa1d86fbc21275611138bf96aa9adb1320e5dffd63358e1246e67b06f7da"} Nov 03 21:36:03.094741 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:03.094305 4963 scope.go:122] "RemoveContainer" containerID="da3c6454c3d7a5efabccee7b9189ab13ca04082e0cb15ca672ea225b58a57e3d" Nov 03 21:36:03.308562 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:03.308478 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v"] Nov 03 21:36:03.355385 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:03.355345 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-nvr5v"] Nov 03 21:36:04.900303 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:04.900265 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:04.969051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:04.969017 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1650f263-d416-4ba6-8067-63828b7c64d3" path="/var/lib/kubelet/pods/1650f263-d416-4ba6-8067-63828b7c64d3/volumes" Nov 03 21:36:05.520836 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.520768 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:36:05.602535 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.602493 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istiod-ca-cert\") pod \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " Nov 03 21:36:05.602738 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.602554 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-podinfo\") pod \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " Nov 03 21:36:05.602738 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.602582 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-workload-socket\") pod \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " Nov 03 21:36:05.602738 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.602715 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-credential-socket\") pod \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " Nov 03 21:36:05.602949 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.602835 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-workload-certs\") pod \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " Nov 03 21:36:05.602949 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.602893 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-workload-socket" pod "7b117e23-def9-49c7-a6bd-db4ccb5c1a12" (UID: "7b117e23-def9-49c7-a6bd-db4ccb5c1a12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:05.603077 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.602993 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-kube-api-access-2q66s\" (UniqueName: \"kubernetes.io/projected/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-kube-api-access-2q66s\") pod \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " Nov 03 21:36:05.603138 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.603049 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-credential-socket" pod "7b117e23-def9-49c7-a6bd-db4ccb5c1a12" (UID: "7b117e23-def9-49c7-a6bd-db4ccb5c1a12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:05.603138 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.603075 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-token\" (UniqueName: \"kubernetes.io/projected/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-token\") pod \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " Nov 03 21:36:05.603138 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.603093 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istiod-ca-cert" pod "7b117e23-def9-49c7-a6bd-db4ccb5c1a12" (UID: "7b117e23-def9-49c7-a6bd-db4ccb5c1a12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:36:05.603138 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.603115 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-workload-certs" pod "7b117e23-def9-49c7-a6bd-db4ccb5c1a12" (UID: "7b117e23-def9-49c7-a6bd-db4ccb5c1a12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:05.603138 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.603128 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-envoy\") pod \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " Nov 03 21:36:05.603401 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.603163 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-data\") pod \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\" (UID: \"7b117e23-def9-49c7-a6bd-db4ccb5c1a12\") " Nov 03 21:36:05.603506 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.603483 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-data" pod "7b117e23-def9-49c7-a6bd-db4ccb5c1a12" (UID: "7b117e23-def9-49c7-a6bd-db4ccb5c1a12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:05.603703 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.603683 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.603779 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.603710 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.603779 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.603725 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.603779 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.603739 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.603779 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.603753 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.605107 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.605081 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-podinfo" pod "7b117e23-def9-49c7-a6bd-db4ccb5c1a12" (UID: "7b117e23-def9-49c7-a6bd-db4ccb5c1a12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:36:05.605385 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.605370 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-envoy" pod "7b117e23-def9-49c7-a6bd-db4ccb5c1a12" (UID: "7b117e23-def9-49c7-a6bd-db4ccb5c1a12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:05.605574 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.605550 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-kube-api-access-2q66s" pod "7b117e23-def9-49c7-a6bd-db4ccb5c1a12" (UID: "7b117e23-def9-49c7-a6bd-db4ccb5c1a12"). InnerVolumeSpecName "kube-api-access-2q66s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:36:05.605656 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.605575 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-token" pod "7b117e23-def9-49c7-a6bd-db4ccb5c1a12" (UID: "7b117e23-def9-49c7-a6bd-db4ccb5c1a12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:36:05.705140 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.705103 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.705140 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.705136 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.705140 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.705145 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:05.705396 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:05.705155 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q66s\" (UniqueName: \"kubernetes.io/projected/7b117e23-def9-49c7-a6bd-db4ccb5c1a12-kube-api-access-2q66s\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:36:06.115657 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:06.115616 4963 generic.go:356] "Generic (PLEG): container finished" podID="7b117e23-def9-49c7-a6bd-db4ccb5c1a12" containerID="a1c35caeb52f410821f40603934300bc2258c6e5e2bd7089952d88dc52b930fd" exitCode=0 Nov 03 21:36:06.116090 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:06.115669 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" Nov 03 21:36:06.116090 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:06.115704 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" event={"ID":"7b117e23-def9-49c7-a6bd-db4ccb5c1a12","Type":"ContainerDied","Data":"a1c35caeb52f410821f40603934300bc2258c6e5e2bd7089952d88dc52b930fd"} Nov 03 21:36:06.116090 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:06.115765 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx" event={"ID":"7b117e23-def9-49c7-a6bd-db4ccb5c1a12","Type":"ContainerDied","Data":"52d63c6954b70563647a66891b693d742d9491eb0e2be96e6659bfdb4b1ec645"} Nov 03 21:36:06.116090 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:06.115789 4963 scope.go:122] "RemoveContainer" containerID="a1c35caeb52f410821f40603934300bc2258c6e5e2bd7089952d88dc52b930fd" Nov 03 21:36:06.127533 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:06.127511 4963 scope.go:122] "RemoveContainer" containerID="a1c35caeb52f410821f40603934300bc2258c6e5e2bd7089952d88dc52b930fd" Nov 03 21:36:06.127856 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:36:06.127834 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c35caeb52f410821f40603934300bc2258c6e5e2bd7089952d88dc52b930fd\": container with ID starting with a1c35caeb52f410821f40603934300bc2258c6e5e2bd7089952d88dc52b930fd not found: ID does not exist" containerID="a1c35caeb52f410821f40603934300bc2258c6e5e2bd7089952d88dc52b930fd" Nov 03 21:36:06.127947 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:06.127863 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c35caeb52f410821f40603934300bc2258c6e5e2bd7089952d88dc52b930fd"} err="failed to get container status \"a1c35caeb52f410821f40603934300bc2258c6e5e2bd7089952d88dc52b930fd\": rpc error: code = NotFound desc = could not find container \"a1c35caeb52f410821f40603934300bc2258c6e5e2bd7089952d88dc52b930fd\": container with ID starting with a1c35caeb52f410821f40603934300bc2258c6e5e2bd7089952d88dc52b930fd not found: ID does not exist" Nov 03 21:36:06.275832 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:06.275798 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx"] Nov 03 21:36:06.321849 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:06.321815 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-48ldx"] Nov 03 21:36:06.970168 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:06.970130 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b117e23-def9-49c7-a6bd-db4ccb5c1a12" path="/var/lib/kubelet/pods/7b117e23-def9-49c7-a6bd-db4ccb5c1a12/volumes" Nov 03 21:36:07.737625 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:07.737592 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:08.685305 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:08.685273 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:08.867181 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:08.867147 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:09.455276 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:09.455242 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:12.527916 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:36:12.527823 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:36:14.011163 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:14.011128 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:18.526663 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:18.526618 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:23.015533 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:23.015500 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:27.824818 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:27.824778 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:32.409118 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:32.409076 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:36.976669 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:36.976592 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:37.562420 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:37.562380 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:40.383511 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:40.383453 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:41.512821 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:41.512780 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:42.528968 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:36:42.528896 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:36:46.067137 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:46.067103 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:50.831854 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:50.831816 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:55.557443 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:55.557410 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:36:56.733569 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:36:56.733534 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:00.276285 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:00.276254 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:04.963161 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:04.963125 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:06.643249 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:06.643214 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:09.610863 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:09.610825 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:12.527983 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:37:12.527940 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:37:14.269278 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:14.269248 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:19.000037 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:18.999993 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:22.194703 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:22.194670 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:23.541775 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:23.541740 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:25.978748 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:25.978718 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:27.249076 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:27.249036 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:28.145555 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:28.145511 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:31.365686 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:31.365653 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:37:31.381504 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:31.381480 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:37:32.644960 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:32.644890 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:37.417719 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:37.417682 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:37.893939 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:37.893874 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:41.911185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:41.911102 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:42.528508 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:37:42.528470 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:37:46.461340 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:46.461304 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:50.520321 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:50.520287 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:51.122280 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:51.122242 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:55.605439 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:55.605395 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:57.175456 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:57.175414 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm"] Nov 03 21:37:57.365565 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:57.365535 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:57.624694 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:57.624666 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:37:57.815153 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:37:57.815125 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:00.603794 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:00.603760 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:02.175780 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:02.175735 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm" podUID="422f25ab-7674-472a-bf7b-69f82b889a28" containerName="payload-processing" containerID="cri-o://790dbe7bbc670d4fb47b3161ef09f456a1b0105b0f2ff00181438233e2b01998" gracePeriod=25 Nov 03 21:38:02.455290 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:02.455259 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm" Nov 03 21:38:02.516714 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:02.516677 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422f25ab-7674-472a-bf7b-69f82b889a28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422f25ab-7674-472a-bf7b-69f82b889a28-plugins-config-volume\") pod \"422f25ab-7674-472a-bf7b-69f82b889a28\" (UID: \"422f25ab-7674-472a-bf7b-69f82b889a28\") " Nov 03 21:38:02.516893 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:02.516834 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422f25ab-7674-472a-bf7b-69f82b889a28-kube-api-access-c98j4\" (UniqueName: \"kubernetes.io/projected/422f25ab-7674-472a-bf7b-69f82b889a28-kube-api-access-c98j4\") pod \"422f25ab-7674-472a-bf7b-69f82b889a28\" (UID: \"422f25ab-7674-472a-bf7b-69f82b889a28\") " Nov 03 21:38:02.517191 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:02.517167 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422f25ab-7674-472a-bf7b-69f82b889a28-plugins-config-volume" pod "422f25ab-7674-472a-bf7b-69f82b889a28" (UID: "422f25ab-7674-472a-bf7b-69f82b889a28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:38:02.517344 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:02.517328 4963 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422f25ab-7674-472a-bf7b-69f82b889a28-plugins-config-volume\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:02.519063 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:02.519036 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422f25ab-7674-472a-bf7b-69f82b889a28-kube-api-access-c98j4" pod "422f25ab-7674-472a-bf7b-69f82b889a28" (UID: "422f25ab-7674-472a-bf7b-69f82b889a28"). InnerVolumeSpecName "kube-api-access-c98j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:38:02.618804 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:02.618764 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c98j4\" (UniqueName: \"kubernetes.io/projected/422f25ab-7674-472a-bf7b-69f82b889a28-kube-api-access-c98j4\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:02.896001 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:02.895963 4963 generic.go:356] "Generic (PLEG): container finished" podID="422f25ab-7674-472a-bf7b-69f82b889a28" containerID="790dbe7bbc670d4fb47b3161ef09f456a1b0105b0f2ff00181438233e2b01998" exitCode=0 Nov 03 21:38:02.896195 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:02.896014 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm" Nov 03 21:38:02.896195 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:02.896046 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm" event={"ID":"422f25ab-7674-472a-bf7b-69f82b889a28","Type":"ContainerDied","Data":"790dbe7bbc670d4fb47b3161ef09f456a1b0105b0f2ff00181438233e2b01998"} Nov 03 21:38:02.896195 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:02.896101 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm" event={"ID":"422f25ab-7674-472a-bf7b-69f82b889a28","Type":"ContainerDied","Data":"9017b1cadd3383413e1e6183fcd9f40e04ebb54c8f4b74a7bd615d31d963fa1a"} Nov 03 21:38:02.896195 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:02.896124 4963 scope.go:122] "RemoveContainer" containerID="790dbe7bbc670d4fb47b3161ef09f456a1b0105b0f2ff00181438233e2b01998" Nov 03 21:38:02.906805 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:02.906784 4963 scope.go:122] "RemoveContainer" containerID="790dbe7bbc670d4fb47b3161ef09f456a1b0105b0f2ff00181438233e2b01998" Nov 03 21:38:02.907149 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:38:02.907123 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790dbe7bbc670d4fb47b3161ef09f456a1b0105b0f2ff00181438233e2b01998\": container with ID starting with 790dbe7bbc670d4fb47b3161ef09f456a1b0105b0f2ff00181438233e2b01998 not found: ID does not exist" containerID="790dbe7bbc670d4fb47b3161ef09f456a1b0105b0f2ff00181438233e2b01998" Nov 03 21:38:02.907233 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:02.907161 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790dbe7bbc670d4fb47b3161ef09f456a1b0105b0f2ff00181438233e2b01998"} err="failed to get container status \"790dbe7bbc670d4fb47b3161ef09f456a1b0105b0f2ff00181438233e2b01998\": rpc error: code = NotFound desc = could not find container \"790dbe7bbc670d4fb47b3161ef09f456a1b0105b0f2ff00181438233e2b01998\": container with ID starting with 790dbe7bbc670d4fb47b3161ef09f456a1b0105b0f2ff00181438233e2b01998 not found: ID does not exist" Nov 03 21:38:03.053706 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:03.053671 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm"] Nov 03 21:38:03.101015 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:03.100976 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b22936-9948895cd-c7csm"] Nov 03 21:38:03.629003 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:03.628969 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:04.968568 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:04.968533 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="422f25ab-7674-472a-bf7b-69f82b889a28" path="/var/lib/kubelet/pods/422f25ab-7674-472a-bf7b-69f82b889a28/volumes" Nov 03 21:38:05.144538 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:05.144506 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:09.754140 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:09.754106 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:12.529119 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:38:12.529079 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:38:14.198567 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:14.198534 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:18.716718 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:18.716685 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:23.412330 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:23.412297 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:26.219811 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:26.219776 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:27.928882 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:27.928847 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:32.534455 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:32.534424 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:37.120737 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:37.120702 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:38.231166 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:38.231132 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:38.414515 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:38.414476 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:41.608697 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:41.608666 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:42.528505 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:38:42.528468 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:38:46.126503 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:46.126463 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n"] Nov 03 21:38:46.126998 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:46.126755 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" podUID="bb35897b-74a4-4d21-99d7-20fbe57e4747" containerName="istio-proxy" containerID="cri-o://d41780cf0da4d4d6058ab32c3cfd9cfb840955ce31be338529c128f1bbf5f93e" gracePeriod=30 Nov 03 21:38:46.156538 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:46.156129 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:46.556754 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:46.556704 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8"] Nov 03 21:38:46.557022 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:46.556958 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" podUID="bb7fbcac-f9dc-4c3c-beba-0582649b2226" containerName="istio-proxy" containerID="cri-o://0517b8b1314bfdeedb7333f0dc6d6c8ad2fb23c575d8140f6e4eb3c2dbedd3b0" gracePeriod=30 Nov 03 21:38:50.700396 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:50.700361 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:51.066810 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.066720 4963 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:38:51.066810 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.066800 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" podUID="bb7fbcac-f9dc-4c3c-beba-0582649b2226" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:38:51.408060 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.408031 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:38:51.526713 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.526680 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-token\" (UniqueName: \"kubernetes.io/projected/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-token\") pod \"bb35897b-74a4-4d21-99d7-20fbe57e4747\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " Nov 03 21:38:51.526948 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.526738 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-workload-socket\") pod \"bb35897b-74a4-4d21-99d7-20fbe57e4747\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " Nov 03 21:38:51.526948 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.526791 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-workload-certs\") pod \"bb35897b-74a4-4d21-99d7-20fbe57e4747\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " Nov 03 21:38:51.526948 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.526818 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-envoy\") pod \"bb35897b-74a4-4d21-99d7-20fbe57e4747\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " Nov 03 21:38:51.526948 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.526945 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb35897b-74a4-4d21-99d7-20fbe57e4747-kube-api-access-jsdbl\" (UniqueName: \"kubernetes.io/projected/bb35897b-74a4-4d21-99d7-20fbe57e4747-kube-api-access-jsdbl\") pod \"bb35897b-74a4-4d21-99d7-20fbe57e4747\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " Nov 03 21:38:51.527194 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.527028 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-podinfo\") pod \"bb35897b-74a4-4d21-99d7-20fbe57e4747\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " Nov 03 21:38:51.527194 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.527076 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb35897b-74a4-4d21-99d7-20fbe57e4747-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb35897b-74a4-4d21-99d7-20fbe57e4747-istiod-ca-cert\") pod \"bb35897b-74a4-4d21-99d7-20fbe57e4747\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " Nov 03 21:38:51.527194 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.527108 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-data\") pod \"bb35897b-74a4-4d21-99d7-20fbe57e4747\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " Nov 03 21:38:51.527194 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.527134 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-workload-certs" pod "bb35897b-74a4-4d21-99d7-20fbe57e4747" (UID: "bb35897b-74a4-4d21-99d7-20fbe57e4747"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:51.527194 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.527165 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-credential-socket\") pod \"bb35897b-74a4-4d21-99d7-20fbe57e4747\" (UID: \"bb35897b-74a4-4d21-99d7-20fbe57e4747\") " Nov 03 21:38:51.527629 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.527500 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-workload-socket" pod "bb35897b-74a4-4d21-99d7-20fbe57e4747" (UID: "bb35897b-74a4-4d21-99d7-20fbe57e4747"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:51.527762 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.527500 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb35897b-74a4-4d21-99d7-20fbe57e4747-istiod-ca-cert" pod "bb35897b-74a4-4d21-99d7-20fbe57e4747" (UID: "bb35897b-74a4-4d21-99d7-20fbe57e4747"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:38:51.527835 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.527745 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-data" pod "bb35897b-74a4-4d21-99d7-20fbe57e4747" (UID: "bb35897b-74a4-4d21-99d7-20fbe57e4747"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:51.528040 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.527983 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-credential-socket" pod "bb35897b-74a4-4d21-99d7-20fbe57e4747" (UID: "bb35897b-74a4-4d21-99d7-20fbe57e4747"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:51.528197 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.528172 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:51.528364 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.528343 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:51.528469 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.528369 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb35897b-74a4-4d21-99d7-20fbe57e4747-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:51.528469 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.528384 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:51.528469 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.528399 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:51.529649 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.529614 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-token" pod "bb35897b-74a4-4d21-99d7-20fbe57e4747" (UID: "bb35897b-74a4-4d21-99d7-20fbe57e4747"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:38:51.529797 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.529701 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-podinfo" pod "bb35897b-74a4-4d21-99d7-20fbe57e4747" (UID: "bb35897b-74a4-4d21-99d7-20fbe57e4747"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:38:51.529797 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.529719 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-envoy" pod "bb35897b-74a4-4d21-99d7-20fbe57e4747" (UID: "bb35897b-74a4-4d21-99d7-20fbe57e4747"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:51.530139 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.530047 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb35897b-74a4-4d21-99d7-20fbe57e4747-kube-api-access-jsdbl" pod "bb35897b-74a4-4d21-99d7-20fbe57e4747" (UID: "bb35897b-74a4-4d21-99d7-20fbe57e4747"). InnerVolumeSpecName "kube-api-access-jsdbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:38:51.630161 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.630084 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsdbl\" (UniqueName: \"kubernetes.io/projected/bb35897b-74a4-4d21-99d7-20fbe57e4747-kube-api-access-jsdbl\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:51.630161 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.630115 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:51.630161 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.630124 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:51.630161 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.630133 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb35897b-74a4-4d21-99d7-20fbe57e4747-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:51.825227 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.825199 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:38:51.933837 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.933751 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-workload-certs\") pod \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " Nov 03 21:38:51.933837 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.933821 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-envoy\") pod \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " Nov 03 21:38:51.934062 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.933902 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istiod-ca-cert\") pod \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " Nov 03 21:38:51.934118 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.934059 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-credential-socket\") pod \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " Nov 03 21:38:51.934183 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.934117 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-podinfo\") pod \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " Nov 03 21:38:51.934183 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.934117 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-workload-certs" pod "bb7fbcac-f9dc-4c3c-beba-0582649b2226" (UID: "bb7fbcac-f9dc-4c3c-beba-0582649b2226"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:51.934183 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.934165 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-workload-socket\") pod \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " Nov 03 21:38:51.934344 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.934202 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-data\") pod \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " Nov 03 21:38:51.934344 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.934233 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-token\" (UniqueName: \"kubernetes.io/projected/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-token\") pod \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " Nov 03 21:38:51.934344 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.934269 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-credential-socket" pod "bb7fbcac-f9dc-4c3c-beba-0582649b2226" (UID: "bb7fbcac-f9dc-4c3c-beba-0582649b2226"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:51.934344 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.934339 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7fbcac-f9dc-4c3c-beba-0582649b2226-kube-api-access-5k6xz\" (UniqueName: \"kubernetes.io/projected/bb7fbcac-f9dc-4c3c-beba-0582649b2226-kube-api-access-5k6xz\") pod \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\" (UID: \"bb7fbcac-f9dc-4c3c-beba-0582649b2226\") " Nov 03 21:38:51.934555 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.934348 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istiod-ca-cert" pod "bb7fbcac-f9dc-4c3c-beba-0582649b2226" (UID: "bb7fbcac-f9dc-4c3c-beba-0582649b2226"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:38:51.934555 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.934468 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-data" pod "bb7fbcac-f9dc-4c3c-beba-0582649b2226" (UID: "bb7fbcac-f9dc-4c3c-beba-0582649b2226"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:51.934817 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.934792 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-workload-socket" pod "bb7fbcac-f9dc-4c3c-beba-0582649b2226" (UID: "bb7fbcac-f9dc-4c3c-beba-0582649b2226"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:51.934957 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.934893 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:51.935036 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.934962 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:51.935036 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.934973 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:51.935036 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.934986 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:51.935036 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.934998 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:51.936481 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.936452 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-podinfo" pod "bb7fbcac-f9dc-4c3c-beba-0582649b2226" (UID: "bb7fbcac-f9dc-4c3c-beba-0582649b2226"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:38:51.936586 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.936567 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7fbcac-f9dc-4c3c-beba-0582649b2226-kube-api-access-5k6xz" pod "bb7fbcac-f9dc-4c3c-beba-0582649b2226" (UID: "bb7fbcac-f9dc-4c3c-beba-0582649b2226"). InnerVolumeSpecName "kube-api-access-5k6xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:38:51.936678 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.936662 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-envoy" pod "bb7fbcac-f9dc-4c3c-beba-0582649b2226" (UID: "bb7fbcac-f9dc-4c3c-beba-0582649b2226"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:38:51.936752 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:51.936735 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-token" pod "bb7fbcac-f9dc-4c3c-beba-0582649b2226" (UID: "bb7fbcac-f9dc-4c3c-beba-0582649b2226"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:38:52.040771 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.040444 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:52.040771 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.040482 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:52.040771 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.040497 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7fbcac-f9dc-4c3c-beba-0582649b2226-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:52.040771 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.040511 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k6xz\" (UniqueName: \"kubernetes.io/projected/bb7fbcac-f9dc-4c3c-beba-0582649b2226-kube-api-access-5k6xz\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:52.208994 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.208957 4963 generic.go:356] "Generic (PLEG): container finished" podID="bb35897b-74a4-4d21-99d7-20fbe57e4747" containerID="d41780cf0da4d4d6058ab32c3cfd9cfb840955ce31be338529c128f1bbf5f93e" exitCode=0 Nov 03 21:38:52.209182 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.209009 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" Nov 03 21:38:52.209182 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.209038 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" event={"ID":"bb35897b-74a4-4d21-99d7-20fbe57e4747","Type":"ContainerDied","Data":"d41780cf0da4d4d6058ab32c3cfd9cfb840955ce31be338529c128f1bbf5f93e"} Nov 03 21:38:52.209182 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.209101 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n" event={"ID":"bb35897b-74a4-4d21-99d7-20fbe57e4747","Type":"ContainerDied","Data":"ecd6be01f76169b7bc3c3cde2ab1ff4d72d360067c7bfb37ccff721bf475f8e4"} Nov 03 21:38:52.209182 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.209137 4963 scope.go:122] "RemoveContainer" containerID="d41780cf0da4d4d6058ab32c3cfd9cfb840955ce31be338529c128f1bbf5f93e" Nov 03 21:38:52.210481 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.210461 4963 generic.go:356] "Generic (PLEG): container finished" podID="bb7fbcac-f9dc-4c3c-beba-0582649b2226" containerID="0517b8b1314bfdeedb7333f0dc6d6c8ad2fb23c575d8140f6e4eb3c2dbedd3b0" exitCode=0 Nov 03 21:38:52.210572 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.210508 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" Nov 03 21:38:52.210572 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.210554 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" event={"ID":"bb7fbcac-f9dc-4c3c-beba-0582649b2226","Type":"ContainerDied","Data":"0517b8b1314bfdeedb7333f0dc6d6c8ad2fb23c575d8140f6e4eb3c2dbedd3b0"} Nov 03 21:38:52.210659 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.210602 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8" event={"ID":"bb7fbcac-f9dc-4c3c-beba-0582649b2226","Type":"ContainerDied","Data":"fa87cbd1c5fb18e4d285529abb4742d699e89d6a3efb2f090784952cfdb27316"} Nov 03 21:38:52.220198 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.220176 4963 scope.go:122] "RemoveContainer" containerID="d41780cf0da4d4d6058ab32c3cfd9cfb840955ce31be338529c128f1bbf5f93e" Nov 03 21:38:52.220551 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:38:52.220510 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d41780cf0da4d4d6058ab32c3cfd9cfb840955ce31be338529c128f1bbf5f93e\": container with ID starting with d41780cf0da4d4d6058ab32c3cfd9cfb840955ce31be338529c128f1bbf5f93e not found: ID does not exist" containerID="d41780cf0da4d4d6058ab32c3cfd9cfb840955ce31be338529c128f1bbf5f93e" Nov 03 21:38:52.220642 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.220557 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41780cf0da4d4d6058ab32c3cfd9cfb840955ce31be338529c128f1bbf5f93e"} err="failed to get container status \"d41780cf0da4d4d6058ab32c3cfd9cfb840955ce31be338529c128f1bbf5f93e\": rpc error: code = NotFound desc = could not find container \"d41780cf0da4d4d6058ab32c3cfd9cfb840955ce31be338529c128f1bbf5f93e\": container with ID starting with d41780cf0da4d4d6058ab32c3cfd9cfb840955ce31be338529c128f1bbf5f93e not found: ID does not exist" Nov 03 21:38:52.220642 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.220582 4963 scope.go:122] "RemoveContainer" containerID="0517b8b1314bfdeedb7333f0dc6d6c8ad2fb23c575d8140f6e4eb3c2dbedd3b0" Nov 03 21:38:52.230440 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.230421 4963 scope.go:122] "RemoveContainer" containerID="0517b8b1314bfdeedb7333f0dc6d6c8ad2fb23c575d8140f6e4eb3c2dbedd3b0" Nov 03 21:38:52.230686 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:38:52.230667 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0517b8b1314bfdeedb7333f0dc6d6c8ad2fb23c575d8140f6e4eb3c2dbedd3b0\": container with ID starting with 0517b8b1314bfdeedb7333f0dc6d6c8ad2fb23c575d8140f6e4eb3c2dbedd3b0 not found: ID does not exist" containerID="0517b8b1314bfdeedb7333f0dc6d6c8ad2fb23c575d8140f6e4eb3c2dbedd3b0" Nov 03 21:38:52.230745 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.230695 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0517b8b1314bfdeedb7333f0dc6d6c8ad2fb23c575d8140f6e4eb3c2dbedd3b0"} err="failed to get container status \"0517b8b1314bfdeedb7333f0dc6d6c8ad2fb23c575d8140f6e4eb3c2dbedd3b0\": rpc error: code = NotFound desc = could not find container \"0517b8b1314bfdeedb7333f0dc6d6c8ad2fb23c575d8140f6e4eb3c2dbedd3b0\": container with ID starting with 0517b8b1314bfdeedb7333f0dc6d6c8ad2fb23c575d8140f6e4eb3c2dbedd3b0 not found: ID does not exist" Nov 03 21:38:52.373471 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.370329 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n"] Nov 03 21:38:52.415639 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.415597 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-bgp8n"] Nov 03 21:38:52.480838 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.480759 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8"] Nov 03 21:38:52.486705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.486673 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-hc4c8"] Nov 03 21:38:52.967619 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.967585 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb35897b-74a4-4d21-99d7-20fbe57e4747" path="/var/lib/kubelet/pods/bb35897b-74a4-4d21-99d7-20fbe57e4747/volumes" Nov 03 21:38:52.968092 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:52.968077 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb7fbcac-f9dc-4c3c-beba-0582649b2226" path="/var/lib/kubelet/pods/bb7fbcac-f9dc-4c3c-beba-0582649b2226/volumes" Nov 03 21:38:55.374687 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:55.374653 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:38:58.420864 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:58.420825 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp"] Nov 03 21:38:58.421365 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:38:58.421102 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" podUID="f0afb830-dc6c-40af-b463-3edeffe576e4" containerName="istio-proxy" containerID="cri-o://3ea222969033222b330e02b52f4f049a60c51fe88b6ea0f21daf424dc420d1ca" gracePeriod=30 Nov 03 21:39:00.029770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:00.029735 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:03.366900 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.366862 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:39:03.367321 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.366952 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" podUID="f0afb830-dc6c-40af-b463-3edeffe576e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:39:03.698747 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.698714 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:39:03.779821 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.779786 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-workload-certs\") pod \"f0afb830-dc6c-40af-b463-3edeffe576e4\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " Nov 03 21:39:03.780003 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.779834 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-credential-socket\") pod \"f0afb830-dc6c-40af-b463-3edeffe576e4\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " Nov 03 21:39:03.780003 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.779872 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-envoy\") pod \"f0afb830-dc6c-40af-b463-3edeffe576e4\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " Nov 03 21:39:03.780003 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.779893 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0afb830-dc6c-40af-b463-3edeffe576e4-kube-api-access-flblh\" (UniqueName: \"kubernetes.io/projected/f0afb830-dc6c-40af-b463-3edeffe576e4-kube-api-access-flblh\") pod \"f0afb830-dc6c-40af-b463-3edeffe576e4\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " Nov 03 21:39:03.780003 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.779977 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-podinfo\") pod \"f0afb830-dc6c-40af-b463-3edeffe576e4\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " Nov 03 21:39:03.780257 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.780013 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0afb830-dc6c-40af-b463-3edeffe576e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0afb830-dc6c-40af-b463-3edeffe576e4-istiod-ca-cert\") pod \"f0afb830-dc6c-40af-b463-3edeffe576e4\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " Nov 03 21:39:03.780257 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.780109 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-data\") pod \"f0afb830-dc6c-40af-b463-3edeffe576e4\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " Nov 03 21:39:03.780257 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.780146 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-token\" (UniqueName: \"kubernetes.io/projected/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-token\") pod \"f0afb830-dc6c-40af-b463-3edeffe576e4\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " Nov 03 21:39:03.780257 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.780173 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-workload-socket\") pod \"f0afb830-dc6c-40af-b463-3edeffe576e4\" (UID: \"f0afb830-dc6c-40af-b463-3edeffe576e4\") " Nov 03 21:39:03.780257 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.780181 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-credential-socket" pod "f0afb830-dc6c-40af-b463-3edeffe576e4" (UID: "f0afb830-dc6c-40af-b463-3edeffe576e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:39:03.780520 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.780318 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-workload-certs" pod "f0afb830-dc6c-40af-b463-3edeffe576e4" (UID: "f0afb830-dc6c-40af-b463-3edeffe576e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:39:03.780579 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.780536 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-workload-socket" pod "f0afb830-dc6c-40af-b463-3edeffe576e4" (UID: "f0afb830-dc6c-40af-b463-3edeffe576e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:39:03.780579 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.780562 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:03.780692 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.780583 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:03.780899 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.780873 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-data" pod "f0afb830-dc6c-40af-b463-3edeffe576e4" (UID: "f0afb830-dc6c-40af-b463-3edeffe576e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:39:03.781032 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.781007 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0afb830-dc6c-40af-b463-3edeffe576e4-istiod-ca-cert" pod "f0afb830-dc6c-40af-b463-3edeffe576e4" (UID: "f0afb830-dc6c-40af-b463-3edeffe576e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:39:03.782821 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.782791 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-podinfo" pod "f0afb830-dc6c-40af-b463-3edeffe576e4" (UID: "f0afb830-dc6c-40af-b463-3edeffe576e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:39:03.783126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.783098 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-envoy" pod "f0afb830-dc6c-40af-b463-3edeffe576e4" (UID: "f0afb830-dc6c-40af-b463-3edeffe576e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:39:03.783219 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.783121 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0afb830-dc6c-40af-b463-3edeffe576e4-kube-api-access-flblh" pod "f0afb830-dc6c-40af-b463-3edeffe576e4" (UID: "f0afb830-dc6c-40af-b463-3edeffe576e4"). InnerVolumeSpecName "kube-api-access-flblh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:39:03.783219 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.783149 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-token" pod "f0afb830-dc6c-40af-b463-3edeffe576e4" (UID: "f0afb830-dc6c-40af-b463-3edeffe576e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:39:03.881617 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.881579 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:03.881617 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.881619 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:03.881829 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.881635 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:03.881829 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.881650 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:03.881829 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.881665 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flblh\" (UniqueName: \"kubernetes.io/projected/f0afb830-dc6c-40af-b463-3edeffe576e4-kube-api-access-flblh\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:03.881829 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.881680 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0afb830-dc6c-40af-b463-3edeffe576e4-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:03.881829 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:03.881692 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0afb830-dc6c-40af-b463-3edeffe576e4-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:04.298583 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:04.298549 4963 generic.go:356] "Generic (PLEG): container finished" podID="f0afb830-dc6c-40af-b463-3edeffe576e4" containerID="3ea222969033222b330e02b52f4f049a60c51fe88b6ea0f21daf424dc420d1ca" exitCode=0 Nov 03 21:39:04.298800 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:04.298616 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" Nov 03 21:39:04.298800 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:04.298638 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" event={"ID":"f0afb830-dc6c-40af-b463-3edeffe576e4","Type":"ContainerDied","Data":"3ea222969033222b330e02b52f4f049a60c51fe88b6ea0f21daf424dc420d1ca"} Nov 03 21:39:04.298800 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:04.298705 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp" event={"ID":"f0afb830-dc6c-40af-b463-3edeffe576e4","Type":"ContainerDied","Data":"b315a69ffc637253de667358bd79c29add8076cae80ae73eebff69ffd18b6c70"} Nov 03 21:39:04.298800 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:04.298737 4963 scope.go:122] "RemoveContainer" containerID="3ea222969033222b330e02b52f4f049a60c51fe88b6ea0f21daf424dc420d1ca" Nov 03 21:39:04.309722 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:04.309701 4963 scope.go:122] "RemoveContainer" containerID="3ea222969033222b330e02b52f4f049a60c51fe88b6ea0f21daf424dc420d1ca" Nov 03 21:39:04.310035 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:39:04.310015 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea222969033222b330e02b52f4f049a60c51fe88b6ea0f21daf424dc420d1ca\": container with ID starting with 3ea222969033222b330e02b52f4f049a60c51fe88b6ea0f21daf424dc420d1ca not found: ID does not exist" containerID="3ea222969033222b330e02b52f4f049a60c51fe88b6ea0f21daf424dc420d1ca" Nov 03 21:39:04.310113 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:04.310043 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea222969033222b330e02b52f4f049a60c51fe88b6ea0f21daf424dc420d1ca"} err="failed to get container status \"3ea222969033222b330e02b52f4f049a60c51fe88b6ea0f21daf424dc420d1ca\": rpc error: code = NotFound desc = could not find container \"3ea222969033222b330e02b52f4f049a60c51fe88b6ea0f21daf424dc420d1ca\": container with ID starting with 3ea222969033222b330e02b52f4f049a60c51fe88b6ea0f21daf424dc420d1ca not found: ID does not exist" Nov 03 21:39:04.474411 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:04.474373 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp"] Nov 03 21:39:04.480971 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:04.480941 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-gnvkp"] Nov 03 21:39:04.533885 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:04.533847 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:04.968725 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:04.968684 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0afb830-dc6c-40af-b463-3edeffe576e4" path="/var/lib/kubelet/pods/f0afb830-dc6c-40af-b463-3edeffe576e4/volumes" Nov 03 21:39:09.122439 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:09.122400 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:10.287214 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.287133 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz"] Nov 03 21:39:10.287751 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.287731 4963 cpu_manager.go:414] "containerMap: removing container" podUID="bb7fbcac-f9dc-4c3c-beba-0582649b2226" containerName="istio-proxy" Nov 03 21:39:10.287806 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.287753 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7fbcac-f9dc-4c3c-beba-0582649b2226" containerName="istio-proxy" Nov 03 21:39:10.287806 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.287771 4963 cpu_manager.go:414] "containerMap: removing container" podUID="f0afb830-dc6c-40af-b463-3edeffe576e4" containerName="istio-proxy" Nov 03 21:39:10.287806 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.287777 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0afb830-dc6c-40af-b463-3edeffe576e4" containerName="istio-proxy" Nov 03 21:39:10.287806 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.287797 4963 cpu_manager.go:414] "containerMap: removing container" podUID="bb35897b-74a4-4d21-99d7-20fbe57e4747" containerName="istio-proxy" Nov 03 21:39:10.287806 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.287803 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb35897b-74a4-4d21-99d7-20fbe57e4747" containerName="istio-proxy" Nov 03 21:39:10.287976 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.287831 4963 cpu_manager.go:414] "containerMap: removing container" podUID="422f25ab-7674-472a-bf7b-69f82b889a28" containerName="payload-processing" Nov 03 21:39:10.287976 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.287836 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422f25ab-7674-472a-bf7b-69f82b889a28" containerName="payload-processing" Nov 03 21:39:10.287976 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.287847 4963 cpu_manager.go:414] "containerMap: removing container" podUID="7b117e23-def9-49c7-a6bd-db4ccb5c1a12" containerName="istio-proxy" Nov 03 21:39:10.287976 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.287852 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b117e23-def9-49c7-a6bd-db4ccb5c1a12" containerName="istio-proxy" Nov 03 21:39:10.287976 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.287857 4963 cpu_manager.go:414] "containerMap: removing container" podUID="1650f263-d416-4ba6-8067-63828b7c64d3" containerName="istio-proxy" Nov 03 21:39:10.287976 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.287861 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1650f263-d416-4ba6-8067-63828b7c64d3" containerName="istio-proxy" Nov 03 21:39:10.287976 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.287960 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb35897b-74a4-4d21-99d7-20fbe57e4747" containerName="istio-proxy" Nov 03 21:39:10.287976 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.287974 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="422f25ab-7674-472a-bf7b-69f82b889a28" containerName="payload-processing" Nov 03 21:39:10.287976 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.287984 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b117e23-def9-49c7-a6bd-db4ccb5c1a12" containerName="istio-proxy" Nov 03 21:39:10.288257 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.287990 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="1650f263-d416-4ba6-8067-63828b7c64d3" containerName="istio-proxy" Nov 03 21:39:10.288257 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.287997 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7fbcac-f9dc-4c3c-beba-0582649b2226" containerName="istio-proxy" Nov 03 21:39:10.288257 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.288004 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0afb830-dc6c-40af-b463-3edeffe576e4" containerName="istio-proxy" Nov 03 21:39:10.292881 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.292861 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.352461 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.352432 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-home\") pod \"auth-test-model-714979-kserve-6bd4f49845-tk9xz\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.352644 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.352563 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a30a91-1bf3-422b-9f88-3023c4431e1d-tls-certs\") pod \"auth-test-model-714979-kserve-6bd4f49845-tk9xz\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.352716 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.352660 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44g9j\" (UniqueName: \"kubernetes.io/projected/02a30a91-1bf3-422b-9f88-3023c4431e1d-kube-api-access-44g9j\") pod \"auth-test-model-714979-kserve-6bd4f49845-tk9xz\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.352716 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.352706 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-model-cache\") pod \"auth-test-model-714979-kserve-6bd4f49845-tk9xz\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.352833 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.352740 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-dshm\") pod \"auth-test-model-714979-kserve-6bd4f49845-tk9xz\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.352993 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.352971 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-tmp-dir\") pod \"auth-test-model-714979-kserve-6bd4f49845-tk9xz\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.390201 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.390160 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-714979\"/\"kube-root-ca.crt\"" Nov 03 21:39:10.390201 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.390177 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-714979\"/\"auth-test-model-714979-kserve-self-signed-certs\"" Nov 03 21:39:10.390402 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.390204 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-714979\"/\"default-dockercfg-7hhh4\"" Nov 03 21:39:10.390402 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.390265 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-714979\"/\"openshift-service-ca.crt\"" Nov 03 21:39:10.393053 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.393028 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz"] Nov 03 21:39:10.421237 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.421207 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:10.453830 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.453758 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-home\") pod \"auth-test-model-714979-kserve-6bd4f49845-tk9xz\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.453830 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.453835 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a30a91-1bf3-422b-9f88-3023c4431e1d-tls-certs\") pod \"auth-test-model-714979-kserve-6bd4f49845-tk9xz\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.454139 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.453875 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44g9j\" (UniqueName: \"kubernetes.io/projected/02a30a91-1bf3-422b-9f88-3023c4431e1d-kube-api-access-44g9j\") pod \"auth-test-model-714979-kserve-6bd4f49845-tk9xz\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.454139 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.453937 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-model-cache\") pod \"auth-test-model-714979-kserve-6bd4f49845-tk9xz\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.454139 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.453958 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-dshm\") pod \"auth-test-model-714979-kserve-6bd4f49845-tk9xz\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.454139 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.454040 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-tmp-dir\") pod \"auth-test-model-714979-kserve-6bd4f49845-tk9xz\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.454367 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.454315 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-home\") pod \"auth-test-model-714979-kserve-6bd4f49845-tk9xz\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.454367 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.454326 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-model-cache\") pod \"auth-test-model-714979-kserve-6bd4f49845-tk9xz\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.454442 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.454423 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-tmp-dir\") pod \"auth-test-model-714979-kserve-6bd4f49845-tk9xz\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.456403 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.456381 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-dshm\") pod \"auth-test-model-714979-kserve-6bd4f49845-tk9xz\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.456637 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.456618 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a30a91-1bf3-422b-9f88-3023c4431e1d-tls-certs\") pod \"auth-test-model-714979-kserve-6bd4f49845-tk9xz\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.512509 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.512478 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44g9j\" (UniqueName: \"kubernetes.io/projected/02a30a91-1bf3-422b-9f88-3023c4431e1d-kube-api-access-44g9j\") pod \"auth-test-model-714979-kserve-6bd4f49845-tk9xz\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.604942 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.604829 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:10.989173 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:10.989138 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz"] Nov 03 21:39:10.994226 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:39:10.994191 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02a30a91_1bf3_422b_9f88_3023c4431e1d.slice/crio-ba64281b21aec3eb895b597800f3cb6b0cc71e63bb753ed77bd2b947f6a0f9b5 WatchSource:0}: Error finding container ba64281b21aec3eb895b597800f3cb6b0cc71e63bb753ed77bd2b947f6a0f9b5: Status 404 returned error can't find the container with id ba64281b21aec3eb895b597800f3cb6b0cc71e63bb753ed77bd2b947f6a0f9b5 Nov 03 21:39:11.347656 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:11.347553 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" event={"ID":"02a30a91-1bf3-422b-9f88-3023c4431e1d","Type":"ContainerStarted","Data":"5b18349dde0652af894cb28dd9f388d8aa9b5593f847275ab567224ed0062439"} Nov 03 21:39:11.347656 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:11.347609 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" event={"ID":"02a30a91-1bf3-422b-9f88-3023c4431e1d","Type":"ContainerStarted","Data":"ba64281b21aec3eb895b597800f3cb6b0cc71e63bb753ed77bd2b947f6a0f9b5"} Nov 03 21:39:11.459284 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:11.459222 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" podStartSLOduration=1.459201527 podStartE2EDuration="1.459201527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:11.45848808 +0000 UTC m=+3100.618560353" watchObservedRunningTime="2026-11-03 21:39:11.459201527 +0000 UTC m=+3100.619273798" Nov 03 21:39:11.605682 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:11.605591 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:12.528934 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:39:12.528878 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:39:13.718556 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:13.718501 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:18.235878 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:18.235842 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:20.605386 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:20.605348 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:20.618102 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:20.618072 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:21.420672 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:21.420639 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:39:22.701929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:22.701870 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:27.214859 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:27.214818 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:29.263659 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:29.263623 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv"] Nov 03 21:39:29.302783 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:29.302747 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:30.031380 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:30.031343 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9"] Nov 03 21:39:30.031624 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:30.031567 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" podUID="a175a685-3ef1-4d0e-ac80-ad2572d87bde" containerName="istio-proxy" containerID="cri-o://d95f78a525895e6c50f051243e2a23c96885b9e449d17277a9c8274658e95791" gracePeriod=30 Nov 03 21:39:31.744580 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:31.744542 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:32.265447 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:32.265408 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29"] Nov 03 21:39:32.351326 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:32.351247 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:35.325037 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.324969 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:39:35.347170 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.347137 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-envoy\") pod \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " Nov 03 21:39:35.347376 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.347188 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-token\" (UniqueName: \"kubernetes.io/projected/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-token\") pod \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " Nov 03 21:39:35.347376 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.347247 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-podinfo\") pod \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " Nov 03 21:39:35.347376 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.347273 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-workload-socket\") pod \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " Nov 03 21:39:35.347376 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.347292 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-credential-socket\") pod \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " Nov 03 21:39:35.347610 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.347437 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-data\") pod \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " Nov 03 21:39:35.347610 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.347525 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istiod-ca-cert\") pod \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " Nov 03 21:39:35.347610 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.347597 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a175a685-3ef1-4d0e-ac80-ad2572d87bde-kube-api-access-vrxmq\" (UniqueName: \"kubernetes.io/projected/a175a685-3ef1-4d0e-ac80-ad2572d87bde-kube-api-access-vrxmq\") pod \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " Nov 03 21:39:35.347761 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.347625 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-workload-certs\") pod \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\" (UID: \"a175a685-3ef1-4d0e-ac80-ad2572d87bde\") " Nov 03 21:39:35.348121 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.348090 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istiod-ca-cert" pod "a175a685-3ef1-4d0e-ac80-ad2572d87bde" (UID: "a175a685-3ef1-4d0e-ac80-ad2572d87bde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:39:35.348236 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.348203 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-workload-socket" pod "a175a685-3ef1-4d0e-ac80-ad2572d87bde" (UID: "a175a685-3ef1-4d0e-ac80-ad2572d87bde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:39:35.348321 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.348256 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-data" pod "a175a685-3ef1-4d0e-ac80-ad2572d87bde" (UID: "a175a685-3ef1-4d0e-ac80-ad2572d87bde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:39:35.348321 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.348288 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-workload-certs" pod "a175a685-3ef1-4d0e-ac80-ad2572d87bde" (UID: "a175a685-3ef1-4d0e-ac80-ad2572d87bde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:39:35.348498 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.348474 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-credential-socket" pod "a175a685-3ef1-4d0e-ac80-ad2572d87bde" (UID: "a175a685-3ef1-4d0e-ac80-ad2572d87bde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:39:35.350085 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.350038 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-envoy" pod "a175a685-3ef1-4d0e-ac80-ad2572d87bde" (UID: "a175a685-3ef1-4d0e-ac80-ad2572d87bde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:39:35.350447 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.350411 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-podinfo" pod "a175a685-3ef1-4d0e-ac80-ad2572d87bde" (UID: "a175a685-3ef1-4d0e-ac80-ad2572d87bde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:39:35.350610 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.350575 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-token" pod "a175a685-3ef1-4d0e-ac80-ad2572d87bde" (UID: "a175a685-3ef1-4d0e-ac80-ad2572d87bde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:39:35.350719 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.350655 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a175a685-3ef1-4d0e-ac80-ad2572d87bde-kube-api-access-vrxmq" pod "a175a685-3ef1-4d0e-ac80-ad2572d87bde" (UID: "a175a685-3ef1-4d0e-ac80-ad2572d87bde"). InnerVolumeSpecName "kube-api-access-vrxmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:39:35.449836 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.449763 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:35.449836 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.449826 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrxmq\" (UniqueName: \"kubernetes.io/projected/a175a685-3ef1-4d0e-ac80-ad2572d87bde-kube-api-access-vrxmq\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:35.449836 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.449843 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:35.450162 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.449856 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:35.450162 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.449869 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:35.450162 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.449884 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:35.450162 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.449896 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:35.450162 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.449947 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:35.450162 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.449959 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a175a685-3ef1-4d0e-ac80-ad2572d87bde-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:35.504753 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.504705 4963 generic.go:356] "Generic (PLEG): container finished" podID="a175a685-3ef1-4d0e-ac80-ad2572d87bde" containerID="d95f78a525895e6c50f051243e2a23c96885b9e449d17277a9c8274658e95791" exitCode=0 Nov 03 21:39:35.504977 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.504783 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" Nov 03 21:39:35.504977 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.504895 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" event={"ID":"a175a685-3ef1-4d0e-ac80-ad2572d87bde","Type":"ContainerDied","Data":"d95f78a525895e6c50f051243e2a23c96885b9e449d17277a9c8274658e95791"} Nov 03 21:39:35.504977 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.504961 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9" event={"ID":"a175a685-3ef1-4d0e-ac80-ad2572d87bde","Type":"ContainerDied","Data":"dcdf9d570917f9e63e607c756d3226a1d4eab235fb0640e2aaec6b6db7565796"} Nov 03 21:39:35.505158 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.504989 4963 scope.go:122] "RemoveContainer" containerID="d95f78a525895e6c50f051243e2a23c96885b9e449d17277a9c8274658e95791" Nov 03 21:39:35.518014 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.517987 4963 scope.go:122] "RemoveContainer" containerID="d95f78a525895e6c50f051243e2a23c96885b9e449d17277a9c8274658e95791" Nov 03 21:39:35.518381 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:39:35.518357 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d95f78a525895e6c50f051243e2a23c96885b9e449d17277a9c8274658e95791\": container with ID starting with d95f78a525895e6c50f051243e2a23c96885b9e449d17277a9c8274658e95791 not found: ID does not exist" containerID="d95f78a525895e6c50f051243e2a23c96885b9e449d17277a9c8274658e95791" Nov 03 21:39:35.518475 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.518394 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d95f78a525895e6c50f051243e2a23c96885b9e449d17277a9c8274658e95791"} err="failed to get container status \"d95f78a525895e6c50f051243e2a23c96885b9e449d17277a9c8274658e95791\": rpc error: code = NotFound desc = could not find container \"d95f78a525895e6c50f051243e2a23c96885b9e449d17277a9c8274658e95791\": container with ID starting with d95f78a525895e6c50f051243e2a23c96885b9e449d17277a9c8274658e95791 not found: ID does not exist" Nov 03 21:39:35.675009 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.674894 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9"] Nov 03 21:39:35.675009 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:35.674979 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-q4hj9"] Nov 03 21:39:36.415702 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:36.415668 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:36.969158 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:36.969122 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a175a685-3ef1-4d0e-ac80-ad2572d87bde" path="/var/lib/kubelet/pods/a175a685-3ef1-4d0e-ac80-ad2572d87bde/volumes" Nov 03 21:39:38.573104 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:38.573068 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:40.936416 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:40.936379 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:42.528124 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:39:42.528084 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:39:44.284696 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.284656 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" podUID="ca06cf22-420e-4bde-924a-13c7835a6593" containerName="main" containerID="cri-o://f69b6b3a23f74bc6ec96acaed26acf1e98723f38f11ef7fdc751482b5ba0e678" gracePeriod=45 Nov 03 21:39:44.557626 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.557556 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:39:44.565118 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.565082 4963 generic.go:356] "Generic (PLEG): container finished" podID="ca06cf22-420e-4bde-924a-13c7835a6593" containerID="f69b6b3a23f74bc6ec96acaed26acf1e98723f38f11ef7fdc751482b5ba0e678" exitCode=0 Nov 03 21:39:44.565283 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.565138 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" Nov 03 21:39:44.565283 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.565165 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" event={"ID":"ca06cf22-420e-4bde-924a-13c7835a6593","Type":"ContainerDied","Data":"f69b6b3a23f74bc6ec96acaed26acf1e98723f38f11ef7fdc751482b5ba0e678"} Nov 03 21:39:44.565283 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.565211 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv" event={"ID":"ca06cf22-420e-4bde-924a-13c7835a6593","Type":"ContainerDied","Data":"e7e01ab8526431bc7a7de35f192336751e31b5debf62187c8567ba47f4e81232"} Nov 03 21:39:44.565283 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.565236 4963 scope.go:122] "RemoveContainer" containerID="f69b6b3a23f74bc6ec96acaed26acf1e98723f38f11ef7fdc751482b5ba0e678" Nov 03 21:39:44.574964 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.574942 4963 scope.go:122] "RemoveContainer" containerID="f69b6b3a23f74bc6ec96acaed26acf1e98723f38f11ef7fdc751482b5ba0e678" Nov 03 21:39:44.575281 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:39:44.575263 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69b6b3a23f74bc6ec96acaed26acf1e98723f38f11ef7fdc751482b5ba0e678\": container with ID starting with f69b6b3a23f74bc6ec96acaed26acf1e98723f38f11ef7fdc751482b5ba0e678 not found: ID does not exist" containerID="f69b6b3a23f74bc6ec96acaed26acf1e98723f38f11ef7fdc751482b5ba0e678" Nov 03 21:39:44.575339 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.575290 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69b6b3a23f74bc6ec96acaed26acf1e98723f38f11ef7fdc751482b5ba0e678"} err="failed to get container status \"f69b6b3a23f74bc6ec96acaed26acf1e98723f38f11ef7fdc751482b5ba0e678\": rpc error: code = NotFound desc = could not find container \"f69b6b3a23f74bc6ec96acaed26acf1e98723f38f11ef7fdc751482b5ba0e678\": container with ID starting with f69b6b3a23f74bc6ec96acaed26acf1e98723f38f11ef7fdc751482b5ba0e678 not found: ID does not exist" Nov 03 21:39:44.651655 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.651619 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-tmp-dir\") pod \"ca06cf22-420e-4bde-924a-13c7835a6593\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " Nov 03 21:39:44.651853 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.651695 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca06cf22-420e-4bde-924a-13c7835a6593-kube-api-access-vld56\" (UniqueName: \"kubernetes.io/projected/ca06cf22-420e-4bde-924a-13c7835a6593-kube-api-access-vld56\") pod \"ca06cf22-420e-4bde-924a-13c7835a6593\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " Nov 03 21:39:44.651853 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.651728 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-model-cache\") pod \"ca06cf22-420e-4bde-924a-13c7835a6593\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " Nov 03 21:39:44.652025 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.651874 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca06cf22-420e-4bde-924a-13c7835a6593-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca06cf22-420e-4bde-924a-13c7835a6593-tls-certs\") pod \"ca06cf22-420e-4bde-924a-13c7835a6593\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " Nov 03 21:39:44.652025 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.651986 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-tmp-dir" pod "ca06cf22-420e-4bde-924a-13c7835a6593" (UID: "ca06cf22-420e-4bde-924a-13c7835a6593"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:39:44.652025 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.652000 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-dshm\") pod \"ca06cf22-420e-4bde-924a-13c7835a6593\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " Nov 03 21:39:44.652185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.652038 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-model-cache" pod "ca06cf22-420e-4bde-924a-13c7835a6593" (UID: "ca06cf22-420e-4bde-924a-13c7835a6593"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:39:44.652185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.652090 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-home\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-home\") pod \"ca06cf22-420e-4bde-924a-13c7835a6593\" (UID: \"ca06cf22-420e-4bde-924a-13c7835a6593\") " Nov 03 21:39:44.652395 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.652353 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-home" pod "ca06cf22-420e-4bde-924a-13c7835a6593" (UID: "ca06cf22-420e-4bde-924a-13c7835a6593"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:39:44.652573 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.652557 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:44.652638 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.652579 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:44.652638 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.652594 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:44.654365 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.654340 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca06cf22-420e-4bde-924a-13c7835a6593-kube-api-access-vld56" pod "ca06cf22-420e-4bde-924a-13c7835a6593" (UID: "ca06cf22-420e-4bde-924a-13c7835a6593"). InnerVolumeSpecName "kube-api-access-vld56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:39:44.654461 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.654371 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-dshm" pod "ca06cf22-420e-4bde-924a-13c7835a6593" (UID: "ca06cf22-420e-4bde-924a-13c7835a6593"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:39:44.654461 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.654417 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca06cf22-420e-4bde-924a-13c7835a6593-tls-certs" pod "ca06cf22-420e-4bde-924a-13c7835a6593" (UID: "ca06cf22-420e-4bde-924a-13c7835a6593"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:39:44.754083 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.754043 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca06cf22-420e-4bde-924a-13c7835a6593-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:44.754083 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.754084 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca06cf22-420e-4bde-924a-13c7835a6593-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:44.754304 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.754100 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vld56\" (UniqueName: \"kubernetes.io/projected/ca06cf22-420e-4bde-924a-13c7835a6593-kube-api-access-vld56\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:44.984378 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.984335 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv"] Nov 03 21:39:44.990429 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:44.990394 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c70fca/auth-test-model-c70fca-kserve-56455bd765-pqpkv"] Nov 03 21:39:45.417609 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:45.417570 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:46.968116 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:46.968079 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca06cf22-420e-4bde-924a-13c7835a6593" path="/var/lib/kubelet/pods/ca06cf22-420e-4bde-924a-13c7835a6593/volumes" Nov 03 21:39:47.111869 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.111831 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7490-2a0570"] Nov 03 21:39:47.112439 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.112420 4963 cpu_manager.go:414] "containerMap: removing container" podUID="ca06cf22-420e-4bde-924a-13c7835a6593" containerName="main" Nov 03 21:39:47.112494 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.112443 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca06cf22-420e-4bde-924a-13c7835a6593" containerName="main" Nov 03 21:39:47.112494 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.112483 4963 cpu_manager.go:414] "containerMap: removing container" podUID="a175a685-3ef1-4d0e-ac80-ad2572d87bde" containerName="istio-proxy" Nov 03 21:39:47.112494 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.112489 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a175a685-3ef1-4d0e-ac80-ad2572d87bde" containerName="istio-proxy" Nov 03 21:39:47.112631 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.112620 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca06cf22-420e-4bde-924a-13c7835a6593" containerName="main" Nov 03 21:39:47.112669 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.112638 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="a175a685-3ef1-4d0e-ac80-ad2572d87bde" containerName="istio-proxy" Nov 03 21:39:47.117235 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.117211 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-2a0570" Nov 03 21:39:47.166830 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.166796 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7490-2a0570"] Nov 03 21:39:47.181950 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.181921 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zs2k\" (UniqueName: \"kubernetes.io/projected/422b7558-3dcf-46a4-b6b6-d014d7e71fa1-kube-api-access-8zs2k\") pod \"test-curl-7490-2a0570\" (UID: \"422b7558-3dcf-46a4-b6b6-d014d7e71fa1\") " pod="opendatahub/test-curl-7490-2a0570" Nov 03 21:39:47.210445 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.210411 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7vxf5\"" Nov 03 21:39:47.283622 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.283534 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zs2k\" (UniqueName: \"kubernetes.io/projected/422b7558-3dcf-46a4-b6b6-d014d7e71fa1-kube-api-access-8zs2k\") pod \"test-curl-7490-2a0570\" (UID: \"422b7558-3dcf-46a4-b6b6-d014d7e71fa1\") " pod="opendatahub/test-curl-7490-2a0570" Nov 03 21:39:47.287279 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.287243 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" podUID="276ee27d-7d04-4a02-94cd-2cac042b72a6" containerName="main" containerID="cri-o://f78704e2ae5d67efcde02154f54d5040cb250f992e27bb075cf450accf5a888a" gracePeriod=45 Nov 03 21:39:47.350077 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.350040 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zs2k\" (UniqueName: \"kubernetes.io/projected/422b7558-3dcf-46a4-b6b6-d014d7e71fa1-kube-api-access-8zs2k\") pod \"test-curl-7490-2a0570\" (UID: \"422b7558-3dcf-46a4-b6b6-d014d7e71fa1\") " pod="opendatahub/test-curl-7490-2a0570" Nov 03 21:39:47.429668 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.429633 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-2a0570" Nov 03 21:39:47.586386 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.586352 4963 generic.go:356] "Generic (PLEG): container finished" podID="276ee27d-7d04-4a02-94cd-2cac042b72a6" containerID="f78704e2ae5d67efcde02154f54d5040cb250f992e27bb075cf450accf5a888a" exitCode=0 Nov 03 21:39:47.586569 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.586422 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" event={"ID":"276ee27d-7d04-4a02-94cd-2cac042b72a6","Type":"ContainerDied","Data":"f78704e2ae5d67efcde02154f54d5040cb250f992e27bb075cf450accf5a888a"} Nov 03 21:39:47.586569 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.586471 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" event={"ID":"276ee27d-7d04-4a02-94cd-2cac042b72a6","Type":"ContainerDied","Data":"eb3aba7d5142ed34db44b8801d61b3c8a6a61526e8627ad56c3819cbef247ab5"} Nov 03 21:39:47.586569 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.586487 4963 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb3aba7d5142ed34db44b8801d61b3c8a6a61526e8627ad56c3819cbef247ab5" Nov 03 21:39:47.590351 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.590320 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:39:47.687827 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.687793 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/276ee27d-7d04-4a02-94cd-2cac042b72a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/276ee27d-7d04-4a02-94cd-2cac042b72a6-tls-certs\") pod \"276ee27d-7d04-4a02-94cd-2cac042b72a6\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " Nov 03 21:39:47.688033 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.687950 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-dshm\") pod \"276ee27d-7d04-4a02-94cd-2cac042b72a6\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " Nov 03 21:39:47.688033 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.687980 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/276ee27d-7d04-4a02-94cd-2cac042b72a6-kube-api-access-xr46c\" (UniqueName: \"kubernetes.io/projected/276ee27d-7d04-4a02-94cd-2cac042b72a6-kube-api-access-xr46c\") pod \"276ee27d-7d04-4a02-94cd-2cac042b72a6\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " Nov 03 21:39:47.688033 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.688025 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-tmp-dir\") pod \"276ee27d-7d04-4a02-94cd-2cac042b72a6\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " Nov 03 21:39:47.688221 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.688046 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-model-cache\") pod \"276ee27d-7d04-4a02-94cd-2cac042b72a6\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " Nov 03 21:39:47.688221 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.688122 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-home\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-home\") pod \"276ee27d-7d04-4a02-94cd-2cac042b72a6\" (UID: \"276ee27d-7d04-4a02-94cd-2cac042b72a6\") " Nov 03 21:39:47.688348 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.688294 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-tmp-dir" pod "276ee27d-7d04-4a02-94cd-2cac042b72a6" (UID: "276ee27d-7d04-4a02-94cd-2cac042b72a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:39:47.688432 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.688401 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-model-cache" pod "276ee27d-7d04-4a02-94cd-2cac042b72a6" (UID: "276ee27d-7d04-4a02-94cd-2cac042b72a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:39:47.688432 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.688416 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-home" pod "276ee27d-7d04-4a02-94cd-2cac042b72a6" (UID: "276ee27d-7d04-4a02-94cd-2cac042b72a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:39:47.688617 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.688563 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:47.688617 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.688592 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:47.688617 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.688607 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:47.690366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.690343 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276ee27d-7d04-4a02-94cd-2cac042b72a6-tls-certs" pod "276ee27d-7d04-4a02-94cd-2cac042b72a6" (UID: "276ee27d-7d04-4a02-94cd-2cac042b72a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:39:47.690366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.690358 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-dshm" pod "276ee27d-7d04-4a02-94cd-2cac042b72a6" (UID: "276ee27d-7d04-4a02-94cd-2cac042b72a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:39:47.690510 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.690391 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276ee27d-7d04-4a02-94cd-2cac042b72a6-kube-api-access-xr46c" pod "276ee27d-7d04-4a02-94cd-2cac042b72a6" (UID: "276ee27d-7d04-4a02-94cd-2cac042b72a6"). InnerVolumeSpecName "kube-api-access-xr46c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:39:47.789684 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.789653 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/276ee27d-7d04-4a02-94cd-2cac042b72a6-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:47.789684 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.789683 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr46c\" (UniqueName: \"kubernetes.io/projected/276ee27d-7d04-4a02-94cd-2cac042b72a6-kube-api-access-xr46c\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:47.789939 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.789694 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/276ee27d-7d04-4a02-94cd-2cac042b72a6-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:47.816274 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:47.816192 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7490-2a0570"] Nov 03 21:39:47.821808 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:39:47.821779 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod422b7558_3dcf_46a4_b6b6_d014d7e71fa1.slice/crio-6051ddd092dc0af214658d8c130607391199558880880fd690a65e47c50fd81e WatchSource:0}: Error finding container 6051ddd092dc0af214658d8c130607391199558880880fd690a65e47c50fd81e: Status 404 returned error can't find the container with id 6051ddd092dc0af214658d8c130607391199558880880fd690a65e47c50fd81e Nov 03 21:39:48.595804 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:48.595716 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7490-2a0570" event={"ID":"422b7558-3dcf-46a4-b6b6-d014d7e71fa1","Type":"ContainerStarted","Data":"4e75b2106927859b613287d28a7b1b056e821cbb8c9a2ca96c3725a89e1c15e8"} Nov 03 21:39:48.595804 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:48.595757 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7490-2a0570" event={"ID":"422b7558-3dcf-46a4-b6b6-d014d7e71fa1","Type":"ContainerStarted","Data":"6051ddd092dc0af214658d8c130607391199558880880fd690a65e47c50fd81e"} Nov 03 21:39:48.596274 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:48.595892 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29" Nov 03 21:39:48.705293 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:48.705233 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7490-2a0570" podStartSLOduration=1.705219635 podStartE2EDuration="1.705219635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:48.704127598 +0000 UTC m=+3137.864199869" watchObservedRunningTime="2026-11-03 21:39:48.705219635 +0000 UTC m=+3137.865291906" Nov 03 21:39:48.810741 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:48.810701 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29"] Nov 03 21:39:48.858278 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:48.858195 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a1c634/auth-test-model-a1c634-kserve-5d6469bdb5-nds29"] Nov 03 21:39:48.976597 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:48.976564 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="276ee27d-7d04-4a02-94cd-2cac042b72a6" path="/var/lib/kubelet/pods/276ee27d-7d04-4a02-94cd-2cac042b72a6/volumes" Nov 03 21:39:50.015857 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:50.015827 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:50.195437 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:50.195402 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7490-2a0570"] Nov 03 21:39:50.199509 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:50.199480 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7490-2a0570"] Nov 03 21:39:50.348104 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:50.348028 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7490-1a09ce"] Nov 03 21:39:50.348532 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:50.348519 4963 cpu_manager.go:414] "containerMap: removing container" podUID="422b7558-3dcf-46a4-b6b6-d014d7e71fa1" containerName="test-curl-7490-2a0570" Nov 03 21:39:50.348576 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:50.348533 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422b7558-3dcf-46a4-b6b6-d014d7e71fa1" containerName="test-curl-7490-2a0570" Nov 03 21:39:50.348576 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:50.348557 4963 cpu_manager.go:414] "containerMap: removing container" podUID="276ee27d-7d04-4a02-94cd-2cac042b72a6" containerName="main" Nov 03 21:39:50.348576 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:50.348563 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="276ee27d-7d04-4a02-94cd-2cac042b72a6" containerName="main" Nov 03 21:39:50.348678 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:50.348646 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="276ee27d-7d04-4a02-94cd-2cac042b72a6" containerName="main" Nov 03 21:39:50.348678 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:50.348666 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="422b7558-3dcf-46a4-b6b6-d014d7e71fa1" containerName="test-curl-7490-2a0570" Nov 03 21:39:50.352284 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:50.352264 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-1a09ce" Nov 03 21:39:50.404418 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:50.404376 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7490-1a09ce"] Nov 03 21:39:50.421086 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:50.421048 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd6g7\" (UniqueName: \"kubernetes.io/projected/73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87-kube-api-access-xd6g7\") pod \"test-curl-7490-1a09ce\" (UID: \"73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87\") " pod="opendatahub/test-curl-7490-1a09ce" Nov 03 21:39:50.522405 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:50.522371 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd6g7\" (UniqueName: \"kubernetes.io/projected/73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87-kube-api-access-xd6g7\") pod \"test-curl-7490-1a09ce\" (UID: \"73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87\") " pod="opendatahub/test-curl-7490-1a09ce" Nov 03 21:39:50.580661 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:50.580626 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd6g7\" (UniqueName: \"kubernetes.io/projected/73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87-kube-api-access-xd6g7\") pod \"test-curl-7490-1a09ce\" (UID: \"73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87\") " pod="opendatahub/test-curl-7490-1a09ce" Nov 03 21:39:50.607787 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:50.607692 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7490-2a0570" podUID="422b7558-3dcf-46a4-b6b6-d014d7e71fa1" containerName="test-curl-7490-2a0570" containerID="cri-o://4e75b2106927859b613287d28a7b1b056e821cbb8c9a2ca96c3725a89e1c15e8" gracePeriod=2 Nov 03 21:39:50.656539 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:39:50.656495 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-2a0570\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="422b7558-3dcf-46a4-b6b6-d014d7e71fa1" pod="opendatahub/test-curl-7490-2a0570" Nov 03 21:39:50.664571 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:50.664537 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-1a09ce" Nov 03 21:39:51.011748 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:39:51.011709 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-2a0570\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="422b7558-3dcf-46a4-b6b6-d014d7e71fa1" pod="opendatahub/test-curl-7490-2a0570" Nov 03 21:39:51.039470 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:51.039434 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7490-1a09ce"] Nov 03 21:39:51.044479 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:39:51.044454 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73f4bd5b_35fe_4b93_8d7d_c2a4c2f40f87.slice/crio-d45335fc9ae9ee876fb4651543870bd7922779efab15e353b414b1d1d98de906 WatchSource:0}: Error finding container d45335fc9ae9ee876fb4651543870bd7922779efab15e353b414b1d1d98de906: Status 404 returned error can't find the container with id d45335fc9ae9ee876fb4651543870bd7922779efab15e353b414b1d1d98de906 Nov 03 21:39:51.615552 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:51.615515 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7490-1a09ce" event={"ID":"73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87","Type":"ContainerStarted","Data":"50f5d7c23558b3c91ea03a6b3f47a0ece7897d15d657c33dcbb2d0732f9d8fc3"} Nov 03 21:39:51.615711 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:51.615565 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7490-1a09ce" event={"ID":"73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87","Type":"ContainerStarted","Data":"d45335fc9ae9ee876fb4651543870bd7922779efab15e353b414b1d1d98de906"} Nov 03 21:39:51.729013 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:51.728960 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7490-1a09ce" podStartSLOduration=1.728945545 podStartE2EDuration="1.728945545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:51.727354201 +0000 UTC m=+3140.887426678" watchObservedRunningTime="2026-11-03 21:39:51.728945545 +0000 UTC m=+3140.889017817" Nov 03 21:39:52.855722 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:52.855693 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-2a0570" Nov 03 21:39:52.949031 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:52.949000 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422b7558-3dcf-46a4-b6b6-d014d7e71fa1-kube-api-access-8zs2k\" (UniqueName: \"kubernetes.io/projected/422b7558-3dcf-46a4-b6b6-d014d7e71fa1-kube-api-access-8zs2k\") pod \"422b7558-3dcf-46a4-b6b6-d014d7e71fa1\" (UID: \"422b7558-3dcf-46a4-b6b6-d014d7e71fa1\") " Nov 03 21:39:52.951090 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:52.951067 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422b7558-3dcf-46a4-b6b6-d014d7e71fa1-kube-api-access-8zs2k" pod "422b7558-3dcf-46a4-b6b6-d014d7e71fa1" (UID: "422b7558-3dcf-46a4-b6b6-d014d7e71fa1"). InnerVolumeSpecName "kube-api-access-8zs2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:39:52.967840 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:52.967807 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="422b7558-3dcf-46a4-b6b6-d014d7e71fa1" path="/var/lib/kubelet/pods/422b7558-3dcf-46a4-b6b6-d014d7e71fa1/volumes" Nov 03 21:39:53.050624 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:53.050582 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zs2k\" (UniqueName: \"kubernetes.io/projected/422b7558-3dcf-46a4-b6b6-d014d7e71fa1-kube-api-access-8zs2k\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:53.239607 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:53.239514 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7490-1a09ce"] Nov 03 21:39:53.246542 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:53.246498 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7490-1a09ce"] Nov 03 21:39:53.629238 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:53.629133 4963 generic.go:356] "Generic (PLEG): container finished" podID="422b7558-3dcf-46a4-b6b6-d014d7e71fa1" containerID="4e75b2106927859b613287d28a7b1b056e821cbb8c9a2ca96c3725a89e1c15e8" exitCode=137 Nov 03 21:39:53.629238 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:53.629182 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-2a0570" Nov 03 21:39:53.629238 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:53.629193 4963 scope.go:122] "RemoveContainer" containerID="4e75b2106927859b613287d28a7b1b056e821cbb8c9a2ca96c3725a89e1c15e8" Nov 03 21:39:53.630229 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:53.629455 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7490-1a09ce" podUID="73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87" containerName="test-curl-7490-1a09ce" containerID="cri-o://50f5d7c23558b3c91ea03a6b3f47a0ece7897d15d657c33dcbb2d0732f9d8fc3" gracePeriod=2 Nov 03 21:39:53.638994 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:53.638971 4963 scope.go:122] "RemoveContainer" containerID="4e75b2106927859b613287d28a7b1b056e821cbb8c9a2ca96c3725a89e1c15e8" Nov 03 21:39:53.639288 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:39:53.639264 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e75b2106927859b613287d28a7b1b056e821cbb8c9a2ca96c3725a89e1c15e8\": container with ID starting with 4e75b2106927859b613287d28a7b1b056e821cbb8c9a2ca96c3725a89e1c15e8 not found: ID does not exist" containerID="4e75b2106927859b613287d28a7b1b056e821cbb8c9a2ca96c3725a89e1c15e8" Nov 03 21:39:53.639348 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:53.639299 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e75b2106927859b613287d28a7b1b056e821cbb8c9a2ca96c3725a89e1c15e8"} err="failed to get container status \"4e75b2106927859b613287d28a7b1b056e821cbb8c9a2ca96c3725a89e1c15e8\": rpc error: code = NotFound desc = could not find container \"4e75b2106927859b613287d28a7b1b056e821cbb8c9a2ca96c3725a89e1c15e8\": container with ID starting with 4e75b2106927859b613287d28a7b1b056e821cbb8c9a2ca96c3725a89e1c15e8 not found: ID does not exist" Nov 03 21:39:53.678104 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:39:53.678061 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-1a09ce\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87" pod="opendatahub/test-curl-7490-1a09ce" Nov 03 21:39:53.726882 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:39:53.726842 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-1a09ce\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87" pod="opendatahub/test-curl-7490-1a09ce" Nov 03 21:39:54.618889 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:54.618836 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:54.818856 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:54.818824 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:55.534924 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:55.534878 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:55.717136 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:55.717100 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:55.857449 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:55.857416 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-1a09ce" Nov 03 21:39:55.982542 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:55.982492 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87-kube-api-access-xd6g7\" (UniqueName: \"kubernetes.io/projected/73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87-kube-api-access-xd6g7\") pod \"73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87\" (UID: \"73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87\") " Nov 03 21:39:55.985150 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:55.985113 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87-kube-api-access-xd6g7" pod "73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87" (UID: "73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87"). InnerVolumeSpecName "kube-api-access-xd6g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:39:56.083938 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:56.083826 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd6g7\" (UniqueName: \"kubernetes.io/projected/73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87-kube-api-access-xd6g7\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:56.651739 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:56.651697 4963 generic.go:356] "Generic (PLEG): container finished" podID="73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87" containerID="50f5d7c23558b3c91ea03a6b3f47a0ece7897d15d657c33dcbb2d0732f9d8fc3" exitCode=137 Nov 03 21:39:56.651739 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:56.651735 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-1a09ce" Nov 03 21:39:56.652049 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:56.651793 4963 scope.go:122] "RemoveContainer" containerID="50f5d7c23558b3c91ea03a6b3f47a0ece7897d15d657c33dcbb2d0732f9d8fc3" Nov 03 21:39:56.662986 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:56.662962 4963 scope.go:122] "RemoveContainer" containerID="50f5d7c23558b3c91ea03a6b3f47a0ece7897d15d657c33dcbb2d0732f9d8fc3" Nov 03 21:39:56.663342 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:39:56.663321 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50f5d7c23558b3c91ea03a6b3f47a0ece7897d15d657c33dcbb2d0732f9d8fc3\": container with ID starting with 50f5d7c23558b3c91ea03a6b3f47a0ece7897d15d657c33dcbb2d0732f9d8fc3 not found: ID does not exist" containerID="50f5d7c23558b3c91ea03a6b3f47a0ece7897d15d657c33dcbb2d0732f9d8fc3" Nov 03 21:39:56.663420 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:56.663352 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f5d7c23558b3c91ea03a6b3f47a0ece7897d15d657c33dcbb2d0732f9d8fc3"} err="failed to get container status \"50f5d7c23558b3c91ea03a6b3f47a0ece7897d15d657c33dcbb2d0732f9d8fc3\": rpc error: code = NotFound desc = could not find container \"50f5d7c23558b3c91ea03a6b3f47a0ece7897d15d657c33dcbb2d0732f9d8fc3\": container with ID starting with 50f5d7c23558b3c91ea03a6b3f47a0ece7897d15d657c33dcbb2d0732f9d8fc3 not found: ID does not exist" Nov 03 21:39:56.709896 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:39:56.709852 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-1a09ce\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87" pod="opendatahub/test-curl-7490-1a09ce" Nov 03 21:39:56.978172 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:56.978131 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87" path="/var/lib/kubelet/pods/73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87/volumes" Nov 03 21:39:59.328785 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:59.328748 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:59.517794 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:59.517754 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:39:59.711739 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:39:59.711710 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:01.529841 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:01.529805 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:02.396189 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:02.396145 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:02.584328 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:02.584293 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:03.489348 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:03.489308 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf"] Nov 03 21:40:03.489594 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:03.489564 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" podUID="e7c5e674-acbe-4286-9ea0-8a326eb831b1" containerName="maas-api" containerID="cri-o://021b4764a325f96daef5ddc280c047fbded9af8a06bfca12ca59904c00d0efc0" gracePeriod=30 Nov 03 21:40:03.629266 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:03.629233 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:03.706155 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:03.705958 4963 generic.go:356] "Generic (PLEG): container finished" podID="e7c5e674-acbe-4286-9ea0-8a326eb831b1" containerID="021b4764a325f96daef5ddc280c047fbded9af8a06bfca12ca59904c00d0efc0" exitCode=0 Nov 03 21:40:03.706155 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:03.706041 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" event={"ID":"e7c5e674-acbe-4286-9ea0-8a326eb831b1","Type":"ContainerDied","Data":"021b4764a325f96daef5ddc280c047fbded9af8a06bfca12ca59904c00d0efc0"} Nov 03 21:40:03.766616 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:03.766588 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:40:03.785768 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:03.785723 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7c5e674-acbe-4286-9ea0-8a326eb831b1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7c5e674-acbe-4286-9ea0-8a326eb831b1-metrics-certs\") pod \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\" (UID: \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\") " Nov 03 21:40:03.785964 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:03.785876 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c5e674-acbe-4286-9ea0-8a326eb831b1-kube-api-access-c8vls\" (UniqueName: \"kubernetes.io/projected/e7c5e674-acbe-4286-9ea0-8a326eb831b1-kube-api-access-c8vls\") pod \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\" (UID: \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\") " Nov 03 21:40:03.785964 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:03.785935 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7c5e674-acbe-4286-9ea0-8a326eb831b1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7c5e674-acbe-4286-9ea0-8a326eb831b1-maas-api-tls\") pod \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\" (UID: \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\") " Nov 03 21:40:03.786071 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:03.785970 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c5e674-acbe-4286-9ea0-8a326eb831b1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c5e674-acbe-4286-9ea0-8a326eb831b1-openshift-service-ca\") pod \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\" (UID: \"e7c5e674-acbe-4286-9ea0-8a326eb831b1\") " Nov 03 21:40:03.786442 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:03.786412 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c5e674-acbe-4286-9ea0-8a326eb831b1-openshift-service-ca" pod "e7c5e674-acbe-4286-9ea0-8a326eb831b1" (UID: "e7c5e674-acbe-4286-9ea0-8a326eb831b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:40:03.788207 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:03.788180 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c5e674-acbe-4286-9ea0-8a326eb831b1-kube-api-access-c8vls" pod "e7c5e674-acbe-4286-9ea0-8a326eb831b1" (UID: "e7c5e674-acbe-4286-9ea0-8a326eb831b1"). InnerVolumeSpecName "kube-api-access-c8vls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:40:03.788451 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:03.788429 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c5e674-acbe-4286-9ea0-8a326eb831b1-metrics-certs" pod "e7c5e674-acbe-4286-9ea0-8a326eb831b1" (UID: "e7c5e674-acbe-4286-9ea0-8a326eb831b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:40:03.788535 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:03.788447 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c5e674-acbe-4286-9ea0-8a326eb831b1-maas-api-tls" pod "e7c5e674-acbe-4286-9ea0-8a326eb831b1" (UID: "e7c5e674-acbe-4286-9ea0-8a326eb831b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:40:03.887426 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:03.887388 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7c5e674-acbe-4286-9ea0-8a326eb831b1-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:03.887426 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:03.887422 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8vls\" (UniqueName: \"kubernetes.io/projected/e7c5e674-acbe-4286-9ea0-8a326eb831b1-kube-api-access-c8vls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:03.887426 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:03.887434 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7c5e674-acbe-4286-9ea0-8a326eb831b1-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:03.887736 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:03.887443 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c5e674-acbe-4286-9ea0-8a326eb831b1-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:04.714040 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:04.713993 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" event={"ID":"e7c5e674-acbe-4286-9ea0-8a326eb831b1","Type":"ContainerDied","Data":"2b16da4edb2ff4eb8ab128e5a0f2846344694f8f65ef5fba7174616120fc6568"} Nov 03 21:40:04.714040 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:04.714038 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf" Nov 03 21:40:04.714651 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:04.714066 4963 scope.go:122] "RemoveContainer" containerID="021b4764a325f96daef5ddc280c047fbded9af8a06bfca12ca59904c00d0efc0" Nov 03 21:40:04.871625 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:04.871590 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf"] Nov 03 21:40:04.918922 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:04.918876 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b22936-bcbc4b9f-k7wgf"] Nov 03 21:40:04.968426 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:04.968343 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7c5e674-acbe-4286-9ea0-8a326eb831b1" path="/var/lib/kubelet/pods/e7c5e674-acbe-4286-9ea0-8a326eb831b1/volumes" Nov 03 21:40:05.337671 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:05.337590 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:05.526309 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:05.526272 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:05.832866 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:05.832835 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:06.488840 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:06.488803 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:06.674610 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:06.674580 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:07.107241 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:07.107200 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:07.346620 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:07.346590 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:09.135978 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.135892 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:09.353946 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.353898 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:09.546640 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.546603 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:09.744183 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.744148 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:09.842863 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.842785 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk"] Nov 03 21:40:09.843366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.843345 4963 cpu_manager.go:414] "containerMap: removing container" podUID="e7c5e674-acbe-4286-9ea0-8a326eb831b1" containerName="maas-api" Nov 03 21:40:09.843366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.843364 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c5e674-acbe-4286-9ea0-8a326eb831b1" containerName="maas-api" Nov 03 21:40:09.843567 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.843380 4963 cpu_manager.go:414] "containerMap: removing container" podUID="73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87" containerName="test-curl-7490-1a09ce" Nov 03 21:40:09.843567 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.843386 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87" containerName="test-curl-7490-1a09ce" Nov 03 21:40:09.843567 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.843521 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c5e674-acbe-4286-9ea0-8a326eb831b1" containerName="maas-api" Nov 03 21:40:09.843567 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.843535 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f4bd5b-35fe-4b93-8d7d-c2a4c2f40f87" containerName="test-curl-7490-1a09ce" Nov 03 21:40:09.848596 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.848576 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk" Nov 03 21:40:09.901730 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.901700 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-130a8c\"" Nov 03 21:40:09.904890 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.904861 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk"] Nov 03 21:40:09.951464 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.951430 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks"] Nov 03 21:40:09.955429 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.955400 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks" Nov 03 21:40:09.956519 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.956485 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj6wm\" (UniqueName: \"kubernetes.io/projected/aa6ce46d-6643-423e-b237-e26dc6eb00ca-kube-api-access-bj6wm\") pod \"payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk\" (UID: \"aa6ce46d-6643-423e-b237-e26dc6eb00ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk" Nov 03 21:40:09.956666 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.956553 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6ce46d-6643-423e-b237-e26dc6eb00ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk\" (UID: \"aa6ce46d-6643-423e-b237-e26dc6eb00ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk" Nov 03 21:40:09.998431 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:09.998401 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-130a8c-dockercfg-vznxh\"" Nov 03 21:40:10.007596 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.007554 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks"] Nov 03 21:40:10.057541 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.057497 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj6wm\" (UniqueName: \"kubernetes.io/projected/aa6ce46d-6643-423e-b237-e26dc6eb00ca-kube-api-access-bj6wm\") pod \"payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk\" (UID: \"aa6ce46d-6643-423e-b237-e26dc6eb00ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk" Nov 03 21:40:10.057736 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.057579 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrcvs\" (UniqueName: \"kubernetes.io/projected/ebf2301a-0ad8-449f-a2ff-7888665c112b-kube-api-access-lrcvs\") pod \"payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks\" (UID: \"ebf2301a-0ad8-449f-a2ff-7888665c112b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks" Nov 03 21:40:10.057736 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.057710 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6ce46d-6643-423e-b237-e26dc6eb00ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk\" (UID: \"aa6ce46d-6643-423e-b237-e26dc6eb00ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk" Nov 03 21:40:10.057858 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.057828 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf2301a-0ad8-449f-a2ff-7888665c112b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks\" (UID: \"ebf2301a-0ad8-449f-a2ff-7888665c112b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks" Nov 03 21:40:10.058816 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.058792 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6ce46d-6643-423e-b237-e26dc6eb00ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk\" (UID: \"aa6ce46d-6643-423e-b237-e26dc6eb00ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk" Nov 03 21:40:10.117938 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.117849 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj6wm\" (UniqueName: \"kubernetes.io/projected/aa6ce46d-6643-423e-b237-e26dc6eb00ca-kube-api-access-bj6wm\") pod \"payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk\" (UID: \"aa6ce46d-6643-423e-b237-e26dc6eb00ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk" Nov 03 21:40:10.159186 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.159148 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf2301a-0ad8-449f-a2ff-7888665c112b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks\" (UID: \"ebf2301a-0ad8-449f-a2ff-7888665c112b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks" Nov 03 21:40:10.159595 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.159256 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrcvs\" (UniqueName: \"kubernetes.io/projected/ebf2301a-0ad8-449f-a2ff-7888665c112b-kube-api-access-lrcvs\") pod \"payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks\" (UID: \"ebf2301a-0ad8-449f-a2ff-7888665c112b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks" Nov 03 21:40:10.159881 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.159860 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf2301a-0ad8-449f-a2ff-7888665c112b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks\" (UID: \"ebf2301a-0ad8-449f-a2ff-7888665c112b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks" Nov 03 21:40:10.161951 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.161929 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk" Nov 03 21:40:10.221749 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.221717 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrcvs\" (UniqueName: \"kubernetes.io/projected/ebf2301a-0ad8-449f-a2ff-7888665c112b-kube-api-access-lrcvs\") pod \"payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks\" (UID: \"ebf2301a-0ad8-449f-a2ff-7888665c112b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks" Nov 03 21:40:10.273761 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.273729 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks" Nov 03 21:40:10.300704 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.300674 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:10.539111 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.539077 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk"] Nov 03 21:40:10.542731 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:40:10.542699 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6ce46d_6643_423e_b237_e26dc6eb00ca.slice/crio-d708e271cfd8709b0377da6972f407b52136018c7126f54594d6927a2080c354 WatchSource:0}: Error finding container d708e271cfd8709b0377da6972f407b52136018c7126f54594d6927a2080c354: Status 404 returned error can't find the container with id d708e271cfd8709b0377da6972f407b52136018c7126f54594d6927a2080c354 Nov 03 21:40:10.652481 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.652447 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks"] Nov 03 21:40:10.756456 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.756423 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk" event={"ID":"aa6ce46d-6643-423e-b237-e26dc6eb00ca","Type":"ContainerStarted","Data":"c9e989e6cfe9169bd03c83866d29071a13f6941532f86627fb83471120f49caa"} Nov 03 21:40:10.756642 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.756512 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk" event={"ID":"aa6ce46d-6643-423e-b237-e26dc6eb00ca","Type":"ContainerStarted","Data":"d708e271cfd8709b0377da6972f407b52136018c7126f54594d6927a2080c354"} Nov 03 21:40:10.756713 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.756647 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk" Nov 03 21:40:10.758347 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.758321 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks" event={"ID":"ebf2301a-0ad8-449f-a2ff-7888665c112b","Type":"ContainerStarted","Data":"aab967a31ac532247ed2dbccd0e703b2d785a2c235911d2829359abc43e90296"} Nov 03 21:40:10.758490 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.758358 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks" event={"ID":"ebf2301a-0ad8-449f-a2ff-7888665c112b","Type":"ContainerStarted","Data":"09aaa0af8923632a07a2e15b8566916e348825d85973521c5d63d76c2d25d373"} Nov 03 21:40:10.758490 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.758459 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks" Nov 03 21:40:10.868467 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.868358 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk" podStartSLOduration=1.868343267 podStartE2EDuration="1.868343267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:40:10.866648413 +0000 UTC m=+3160.026720888" watchObservedRunningTime="2026-11-03 21:40:10.868343267 +0000 UTC m=+3160.028415538" Nov 03 21:40:10.980733 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:10.980632 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks" podStartSLOduration=1.98061438 podStartE2EDuration="1.98061438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:40:10.976884388 +0000 UTC m=+3160.136956660" watchObservedRunningTime="2026-11-03 21:40:10.98061438 +0000 UTC m=+3160.140686653" Nov 03 21:40:12.530011 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:40:12.529970 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:40:14.825063 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:14.825029 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:19.411473 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:19.411423 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:21.768423 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:21.768387 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk" Nov 03 21:40:21.768858 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:21.768467 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks" Nov 03 21:40:23.875153 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:23.875117 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:26.256116 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:26.256068 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:28.444511 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:28.444472 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:32.911920 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:32.911881 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:37.388151 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:37.388112 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:38.923378 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:38.923345 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:42.529779 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:40:42.529690 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:40:43.959700 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:43.959663 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:46.268537 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.268492 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl"] Nov 03 21:40:46.274261 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.274229 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.325195 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.325165 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-21475a\"/\"kube-root-ca.crt\"" Nov 03 21:40:46.359591 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.359557 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-21475a\"/\"noten-model-21475a-kserve-self-signed-certs\"" Nov 03 21:40:46.359591 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.359575 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-21475a\"/\"default-dockercfg-9nrvk\"" Nov 03 21:40:46.368300 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.368262 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-home\") pod \"noten-model-21475a-kserve-85895d8945-rd9jl\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.368491 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.368327 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-dshm\") pod \"noten-model-21475a-kserve-85895d8945-rd9jl\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.368491 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.368420 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-model-cache\") pod \"noten-model-21475a-kserve-85895d8945-rd9jl\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.368620 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.368520 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-tmp-dir\") pod \"noten-model-21475a-kserve-85895d8945-rd9jl\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.368620 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.368567 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57ftt\" (UniqueName: \"kubernetes.io/projected/86d96035-bf33-40c5-89aa-887b506aa1f5-kube-api-access-57ftt\") pod \"noten-model-21475a-kserve-85895d8945-rd9jl\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.368736 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.368614 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86d96035-bf33-40c5-89aa-887b506aa1f5-tls-certs\") pod \"noten-model-21475a-kserve-85895d8945-rd9jl\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.372357 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.372336 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-21475a\"/\"openshift-service-ca.crt\"" Nov 03 21:40:46.375363 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.375342 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl"] Nov 03 21:40:46.402369 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.402339 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:46.469860 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.469821 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57ftt\" (UniqueName: \"kubernetes.io/projected/86d96035-bf33-40c5-89aa-887b506aa1f5-kube-api-access-57ftt\") pod \"noten-model-21475a-kserve-85895d8945-rd9jl\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.470072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.469885 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86d96035-bf33-40c5-89aa-887b506aa1f5-tls-certs\") pod \"noten-model-21475a-kserve-85895d8945-rd9jl\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.470072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.469985 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-home\") pod \"noten-model-21475a-kserve-85895d8945-rd9jl\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.470072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.470038 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-dshm\") pod \"noten-model-21475a-kserve-85895d8945-rd9jl\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.470072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.470066 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-model-cache\") pod \"noten-model-21475a-kserve-85895d8945-rd9jl\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.470293 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.470193 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-tmp-dir\") pod \"noten-model-21475a-kserve-85895d8945-rd9jl\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.470524 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.470490 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-home\") pod \"noten-model-21475a-kserve-85895d8945-rd9jl\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.470635 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.470517 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-model-cache\") pod \"noten-model-21475a-kserve-85895d8945-rd9jl\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.470635 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.470557 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-tmp-dir\") pod \"noten-model-21475a-kserve-85895d8945-rd9jl\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.472521 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.472493 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-dshm\") pod \"noten-model-21475a-kserve-85895d8945-rd9jl\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.473258 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.473234 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86d96035-bf33-40c5-89aa-887b506aa1f5-tls-certs\") pod \"noten-model-21475a-kserve-85895d8945-rd9jl\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.528832 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.528741 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57ftt\" (UniqueName: \"kubernetes.io/projected/86d96035-bf33-40c5-89aa-887b506aa1f5-kube-api-access-57ftt\") pod \"noten-model-21475a-kserve-85895d8945-rd9jl\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.585530 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.585479 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:46.586436 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.586412 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:46.970810 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.970778 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl"] Nov 03 21:40:46.976243 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:40:46.976214 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86d96035_bf33_40c5_89aa_887b506aa1f5.slice/crio-46cec5dc8a0e8a90dd8b699d8e6ec5b54d4990edd2c19aab4ffff1619811bbf8 WatchSource:0}: Error finding container 46cec5dc8a0e8a90dd8b699d8e6ec5b54d4990edd2c19aab4ffff1619811bbf8: Status 404 returned error can't find the container with id 46cec5dc8a0e8a90dd8b699d8e6ec5b54d4990edd2c19aab4ffff1619811bbf8 Nov 03 21:40:46.988135 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:46.988105 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" event={"ID":"86d96035-bf33-40c5-89aa-887b506aa1f5","Type":"ContainerStarted","Data":"46cec5dc8a0e8a90dd8b699d8e6ec5b54d4990edd2c19aab4ffff1619811bbf8"} Nov 03 21:40:47.994956 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:47.994893 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" event={"ID":"86d96035-bf33-40c5-89aa-887b506aa1f5","Type":"ContainerStarted","Data":"84684e9e08f9317fe4375f965e98c4c4b90e216f09258dff94e13d8d3d3f23e2"} Nov 03 21:40:48.108248 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:48.108194 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" podStartSLOduration=2.108178973 podStartE2EDuration="2.108178973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:40:48.106519021 +0000 UTC m=+3197.266591293" watchObservedRunningTime="2026-11-03 21:40:48.108178973 +0000 UTC m=+3197.268251244" Nov 03 21:40:48.587093 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:48.587044 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:50.944221 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:50.944179 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:55.432091 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:55.432056 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:56.587311 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:56.587267 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:56.600371 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:56.600338 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:57.061013 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:57.060983 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:40:58.328982 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:58.328948 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:58.962627 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:58.962591 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:59.227188 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:59.227074 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:40:59.912629 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:40:59.912595 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:03.071477 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:03.071440 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:03.256622 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:03.256555 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:04.517223 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:04.517187 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7"] Nov 03 21:41:04.521397 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:04.521375 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:41:04.573434 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:04.573399 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-130a8c-serving-cert\"" Nov 03 21:41:04.623276 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:04.623240 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7"] Nov 03 21:41:04.678623 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:04.678585 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea16191a-c781-4ea4-a759-909277f85cc3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7\" (UID: \"ea16191a-c781-4ea4-a759-909277f85cc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:41:04.678623 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:04.678622 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4t7g\" (UniqueName: \"kubernetes.io/projected/ea16191a-c781-4ea4-a759-909277f85cc3-kube-api-access-g4t7g\") pod \"maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7\" (UID: \"ea16191a-c781-4ea4-a759-909277f85cc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:41:04.678868 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:04.678655 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea16191a-c781-4ea4-a759-909277f85cc3-metrics-certs\") pod \"maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7\" (UID: \"ea16191a-c781-4ea4-a759-909277f85cc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:41:04.679034 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:04.678893 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea16191a-c781-4ea4-a759-909277f85cc3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7\" (UID: \"ea16191a-c781-4ea4-a759-909277f85cc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:41:04.780154 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:04.780061 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea16191a-c781-4ea4-a759-909277f85cc3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7\" (UID: \"ea16191a-c781-4ea4-a759-909277f85cc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:41:04.780154 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:04.780102 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4t7g\" (UniqueName: \"kubernetes.io/projected/ea16191a-c781-4ea4-a759-909277f85cc3-kube-api-access-g4t7g\") pod \"maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7\" (UID: \"ea16191a-c781-4ea4-a759-909277f85cc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:41:04.780154 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:04.780152 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea16191a-c781-4ea4-a759-909277f85cc3-metrics-certs\") pod \"maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7\" (UID: \"ea16191a-c781-4ea4-a759-909277f85cc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:41:04.780409 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:04.780251 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea16191a-c781-4ea4-a759-909277f85cc3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7\" (UID: \"ea16191a-c781-4ea4-a759-909277f85cc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:41:04.780409 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:41:04.780361 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-serving-cert: secret "maas-api-e2e-extmod-a-130a8c-serving-cert" not found Nov 03 21:41:04.780491 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:41:04.780421 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea16191a-c781-4ea4-a759-909277f85cc3-maas-api-tls podName:ea16191a-c781-4ea4-a759-909277f85cc3 nodeName:}" failed. No retries permitted until 2026-11-03 21:41:05.280405673 +0000 UTC m=+3214.440477923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea16191a-c781-4ea4-a759-909277f85cc3-maas-api-tls") pod "maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" (UID: "ea16191a-c781-4ea4-a759-909277f85cc3") : secret "maas-api-e2e-extmod-a-130a8c-serving-cert" not found Nov 03 21:41:04.780880 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:04.780855 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea16191a-c781-4ea4-a759-909277f85cc3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7\" (UID: \"ea16191a-c781-4ea4-a759-909277f85cc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:41:04.782818 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:04.782785 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea16191a-c781-4ea4-a759-909277f85cc3-metrics-certs\") pod \"maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7\" (UID: \"ea16191a-c781-4ea4-a759-909277f85cc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:41:04.798293 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:04.798243 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:04.839576 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:04.839544 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4t7g\" (UniqueName: \"kubernetes.io/projected/ea16191a-c781-4ea4-a759-909277f85cc3-kube-api-access-g4t7g\") pod \"maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7\" (UID: \"ea16191a-c781-4ea4-a759-909277f85cc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:41:04.983695 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:04.983662 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:05.285054 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:05.285019 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea16191a-c781-4ea4-a759-909277f85cc3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7\" (UID: \"ea16191a-c781-4ea4-a759-909277f85cc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:41:05.288019 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:05.287987 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea16191a-c781-4ea4-a759-909277f85cc3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7\" (UID: \"ea16191a-c781-4ea4-a759-909277f85cc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:41:05.436230 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:05.436195 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:41:05.667465 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:05.667383 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:05.816564 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:05.816515 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7"] Nov 03 21:41:05.822056 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:41:05.822016 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea16191a_c781_4ea4_a759_909277f85cc3.slice/crio-c72c60cf543920dd3407612c0d8c4dc91bd9579df0b16fd67c2d9f3de1291607 WatchSource:0}: Error finding container c72c60cf543920dd3407612c0d8c4dc91bd9579df0b16fd67c2d9f3de1291607: Status 404 returned error can't find the container with id c72c60cf543920dd3407612c0d8c4dc91bd9579df0b16fd67c2d9f3de1291607 Nov 03 21:41:05.823672 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:05.823654 4963 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:41:06.105052 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:06.105017 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" event={"ID":"ea16191a-c781-4ea4-a759-909277f85cc3","Type":"ContainerStarted","Data":"c72c60cf543920dd3407612c0d8c4dc91bd9579df0b16fd67c2d9f3de1291607"} Nov 03 21:41:07.112306 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:07.112211 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" event={"ID":"ea16191a-c781-4ea4-a759-909277f85cc3","Type":"ContainerStarted","Data":"573fc2737244542b109417cbfdee6ca74e84a4fa2b6f86d71f4330da79a9a4f0"} Nov 03 21:41:07.112306 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:07.112285 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:41:07.227003 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:07.226947 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" podStartSLOduration=2.350859667 podStartE2EDuration="3.226931572s" totalImagesPullingTime="876.071905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:07.225206966 +0000 UTC m=+3216.385279238" watchObservedRunningTime="2026-11-03 21:41:07.226931572 +0000 UTC m=+3216.387003844" Nov 03 21:41:09.634820 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:09.634784 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:12.527503 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:41:12.527464 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:41:13.124968 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:13.124937 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:41:13.196383 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:13.196351 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:13.381097 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:13.381006 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:14.209495 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:14.209451 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:18.807118 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:18.807084 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:23.307708 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:23.307676 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:24.430306 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:24.430271 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:27.807226 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:27.807191 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:27.992820 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:27.992775 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:28.253770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:28.253740 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:28.434341 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:28.434313 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:31.810635 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:31.810601 4963 scope.go:122] "RemoveContainer" containerID="f78704e2ae5d67efcde02154f54d5040cb250f992e27bb075cf450accf5a888a" Nov 03 21:41:32.295042 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:32.294997 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:33.323513 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:33.323472 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl"] Nov 03 21:41:33.433055 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:33.433019 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:36.849067 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:36.849034 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:39.150123 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:39.150090 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:39.343641 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:39.343607 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:39.528197 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:39.528152 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:41.307615 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:41.307581 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:42.527755 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:41:42.527713 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:41:45.954357 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:45.954325 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:48.344701 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.344631 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" podUID="86d96035-bf33-40c5-89aa-887b506aa1f5" containerName="main" containerID="cri-o://84684e9e08f9317fe4375f965e98c4c4b90e216f09258dff94e13d8d3d3f23e2" gracePeriod=45 Nov 03 21:41:48.645205 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.645176 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:41:48.676302 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.676261 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d96035-bf33-40c5-89aa-887b506aa1f5-kube-api-access-57ftt\" (UniqueName: \"kubernetes.io/projected/86d96035-bf33-40c5-89aa-887b506aa1f5-kube-api-access-57ftt\") pod \"86d96035-bf33-40c5-89aa-887b506aa1f5\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " Nov 03 21:41:48.676500 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.676312 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-model-cache\") pod \"86d96035-bf33-40c5-89aa-887b506aa1f5\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " Nov 03 21:41:48.676500 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.676357 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-tmp-dir\") pod \"86d96035-bf33-40c5-89aa-887b506aa1f5\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " Nov 03 21:41:48.676500 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.676383 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86d96035-bf33-40c5-89aa-887b506aa1f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/86d96035-bf33-40c5-89aa-887b506aa1f5-tls-certs\") pod \"86d96035-bf33-40c5-89aa-887b506aa1f5\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " Nov 03 21:41:48.676500 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.676480 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-home\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-home\") pod \"86d96035-bf33-40c5-89aa-887b506aa1f5\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " Nov 03 21:41:48.676730 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.676633 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-tmp-dir" pod "86d96035-bf33-40c5-89aa-887b506aa1f5" (UID: "86d96035-bf33-40c5-89aa-887b506aa1f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:48.676730 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.676646 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-model-cache" pod "86d96035-bf33-40c5-89aa-887b506aa1f5" (UID: "86d96035-bf33-40c5-89aa-887b506aa1f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:48.676730 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.676659 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-dshm\") pod \"86d96035-bf33-40c5-89aa-887b506aa1f5\" (UID: \"86d96035-bf33-40c5-89aa-887b506aa1f5\") " Nov 03 21:41:48.676889 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.676850 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-home" pod "86d96035-bf33-40c5-89aa-887b506aa1f5" (UID: "86d96035-bf33-40c5-89aa-887b506aa1f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:48.677190 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.677164 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:48.677315 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.677193 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:48.677315 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.677208 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:48.679949 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.679025 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d96035-bf33-40c5-89aa-887b506aa1f5-kube-api-access-57ftt" pod "86d96035-bf33-40c5-89aa-887b506aa1f5" (UID: "86d96035-bf33-40c5-89aa-887b506aa1f5"). InnerVolumeSpecName "kube-api-access-57ftt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:41:48.679949 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.679045 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d96035-bf33-40c5-89aa-887b506aa1f5-tls-certs" pod "86d96035-bf33-40c5-89aa-887b506aa1f5" (UID: "86d96035-bf33-40c5-89aa-887b506aa1f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:41:48.679949 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.679305 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-dshm" pod "86d96035-bf33-40c5-89aa-887b506aa1f5" (UID: "86d96035-bf33-40c5-89aa-887b506aa1f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:48.778648 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.778607 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86d96035-bf33-40c5-89aa-887b506aa1f5-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:48.778648 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.778652 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57ftt\" (UniqueName: \"kubernetes.io/projected/86d96035-bf33-40c5-89aa-887b506aa1f5-kube-api-access-57ftt\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:48.778941 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:48.778670 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86d96035-bf33-40c5-89aa-887b506aa1f5-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:49.392758 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:49.392664 4963 generic.go:356] "Generic (PLEG): container finished" podID="86d96035-bf33-40c5-89aa-887b506aa1f5" containerID="84684e9e08f9317fe4375f965e98c4c4b90e216f09258dff94e13d8d3d3f23e2" exitCode=0 Nov 03 21:41:49.392758 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:49.392725 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" Nov 03 21:41:49.392758 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:49.392747 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" event={"ID":"86d96035-bf33-40c5-89aa-887b506aa1f5","Type":"ContainerDied","Data":"84684e9e08f9317fe4375f965e98c4c4b90e216f09258dff94e13d8d3d3f23e2"} Nov 03 21:41:49.393332 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:49.392793 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl" event={"ID":"86d96035-bf33-40c5-89aa-887b506aa1f5","Type":"ContainerDied","Data":"46cec5dc8a0e8a90dd8b699d8e6ec5b54d4990edd2c19aab4ffff1619811bbf8"} Nov 03 21:41:49.393332 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:49.392811 4963 scope.go:122] "RemoveContainer" containerID="84684e9e08f9317fe4375f965e98c4c4b90e216f09258dff94e13d8d3d3f23e2" Nov 03 21:41:49.402687 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:49.402660 4963 scope.go:122] "RemoveContainer" containerID="84684e9e08f9317fe4375f965e98c4c4b90e216f09258dff94e13d8d3d3f23e2" Nov 03 21:41:49.402992 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:41:49.402969 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84684e9e08f9317fe4375f965e98c4c4b90e216f09258dff94e13d8d3d3f23e2\": container with ID starting with 84684e9e08f9317fe4375f965e98c4c4b90e216f09258dff94e13d8d3d3f23e2 not found: ID does not exist" containerID="84684e9e08f9317fe4375f965e98c4c4b90e216f09258dff94e13d8d3d3f23e2" Nov 03 21:41:49.403032 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:49.403003 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84684e9e08f9317fe4375f965e98c4c4b90e216f09258dff94e13d8d3d3f23e2"} err="failed to get container status \"84684e9e08f9317fe4375f965e98c4c4b90e216f09258dff94e13d8d3d3f23e2\": rpc error: code = NotFound desc = could not find container \"84684e9e08f9317fe4375f965e98c4c4b90e216f09258dff94e13d8d3d3f23e2\": container with ID starting with 84684e9e08f9317fe4375f965e98c4c4b90e216f09258dff94e13d8d3d3f23e2 not found: ID does not exist" Nov 03 21:41:49.549944 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:49.549897 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl"] Nov 03 21:41:49.597150 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:49.597112 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-21475a/noten-model-21475a-kserve-85895d8945-rd9jl"] Nov 03 21:41:50.423123 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:50.423089 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:50.968095 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:50.968051 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d96035-bf33-40c5-89aa-887b506aa1f5" path="/var/lib/kubelet/pods/86d96035-bf33-40c5-89aa-887b506aa1f5/volumes" Nov 03 21:41:53.119289 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:53.119260 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:54.712303 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:54.712263 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:54.922502 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:54.922470 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:41:59.493259 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:41:59.493225 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:03.963565 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:03.963534 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:08.537148 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:08.537115 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:11.107339 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:11.107260 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:12.529138 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:42:12.529102 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:42:12.695455 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:12.695417 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:13.062250 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:13.062205 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:13.247752 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:13.247716 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:13.434691 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:13.434609 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:13.657876 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:13.657843 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:17.879268 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:17.879233 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:18.313627 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:18.313592 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:18.514688 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:18.514657 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:19.833199 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:19.833158 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:20.309167 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:20.309134 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:22.735108 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:22.735074 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:26.139496 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:26.139459 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:26.318791 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:26.318759 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:27.699861 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:27.699822 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:28.981602 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:28.981566 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:29.632232 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:29.632193 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:29.900077 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:29.899982 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:30.750886 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:30.750850 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:30.937865 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:30.937827 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:31.408438 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:31.408411 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:42:31.423445 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:31.423417 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:42:32.326155 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:32.326117 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:32.609093 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:32.609009 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:33.724936 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:33.724881 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:33.927710 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:33.927677 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:34.174476 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:34.174391 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:34.797873 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:34.797821 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:35.024261 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:35.024220 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:35.694659 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:35.694621 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:37.325548 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:37.325516 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:39.625720 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:39.625686 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:39.821741 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:39.821700 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:39.825021 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:39.824988 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7481-ae79e7"] Nov 03 21:42:39.825565 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:39.825548 4963 cpu_manager.go:414] "containerMap: removing container" podUID="86d96035-bf33-40c5-89aa-887b506aa1f5" containerName="main" Nov 03 21:42:39.825565 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:39.825564 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d96035-bf33-40c5-89aa-887b506aa1f5" containerName="main" Nov 03 21:42:39.825699 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:39.825678 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d96035-bf33-40c5-89aa-887b506aa1f5" containerName="main" Nov 03 21:42:39.830267 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:39.830248 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-ae79e7" Nov 03 21:42:39.883532 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:39.883444 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xt52\" (UniqueName: \"kubernetes.io/projected/fb430705-431d-429a-bf48-e004d455e4c5-kube-api-access-2xt52\") pod \"test-curl-7481-ae79e7\" (UID: \"fb430705-431d-429a-bf48-e004d455e4c5\") " pod="opendatahub/test-curl-7481-ae79e7" Nov 03 21:42:39.887842 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:39.887808 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-ae79e7"] Nov 03 21:42:39.930257 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:39.930217 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7vxf5\"" Nov 03 21:42:39.985811 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:39.985772 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xt52\" (UniqueName: \"kubernetes.io/projected/fb430705-431d-429a-bf48-e004d455e4c5-kube-api-access-2xt52\") pod \"test-curl-7481-ae79e7\" (UID: \"fb430705-431d-429a-bf48-e004d455e4c5\") " pod="opendatahub/test-curl-7481-ae79e7" Nov 03 21:42:40.033573 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:40.033517 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:40.044283 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:40.044248 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xt52\" (UniqueName: \"kubernetes.io/projected/fb430705-431d-429a-bf48-e004d455e4c5-kube-api-access-2xt52\") pod \"test-curl-7481-ae79e7\" (UID: \"fb430705-431d-429a-bf48-e004d455e4c5\") " pod="opendatahub/test-curl-7481-ae79e7" Nov 03 21:42:40.145149 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:40.145069 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-ae79e7" Nov 03 21:42:40.524137 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:40.524098 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-ae79e7"] Nov 03 21:42:40.530559 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:42:40.530521 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb430705_431d_429a_bf48_e004d455e4c5.slice/crio-3a27042231a0553569606f51129fa8aa236ca25940f2c350e26d42f84b1c7828 WatchSource:0}: Error finding container 3a27042231a0553569606f51129fa8aa236ca25940f2c350e26d42f84b1c7828: Status 404 returned error can't find the container with id 3a27042231a0553569606f51129fa8aa236ca25940f2c350e26d42f84b1c7828 Nov 03 21:42:40.740465 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:40.740430 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-ae79e7" event={"ID":"fb430705-431d-429a-bf48-e004d455e4c5","Type":"ContainerStarted","Data":"9ae0ee9f9cfdd30b0785a95df1d11c8f8ba438ad9f0dd40e64834d43db0141a7"} Nov 03 21:42:40.740854 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:40.740484 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-ae79e7" event={"ID":"fb430705-431d-429a-bf48-e004d455e4c5","Type":"ContainerStarted","Data":"3a27042231a0553569606f51129fa8aa236ca25940f2c350e26d42f84b1c7828"} Nov 03 21:42:41.477878 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:41.477844 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:41.807924 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:41.807824 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:42.199954 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:42.199872 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:42.371996 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:42.371927 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7481-ae79e7" podStartSLOduration=3.371893232 podStartE2EDuration="3.371893232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:40.852682186 +0000 UTC m=+3310.012754441" watchObservedRunningTime="2026-11-03 21:42:42.371893232 +0000 UTC m=+3311.531965505" Nov 03 21:42:42.373540 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:42.373486 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7481-ae79e7"] Nov 03 21:42:42.378114 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:42.378086 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7481-ae79e7"] Nov 03 21:42:42.528106 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:42:42.528013 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:42:42.755925 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:42.755863 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7481-ae79e7" podUID="fb430705-431d-429a-bf48-e004d455e4c5" containerName="test-curl-7481-ae79e7" containerID="cri-o://9ae0ee9f9cfdd30b0785a95df1d11c8f8ba438ad9f0dd40e64834d43db0141a7" gracePeriod=2 Nov 03 21:42:42.805303 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:42:42.805212 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7481-ae79e7\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="fb430705-431d-429a-bf48-e004d455e4c5" pod="opendatahub/test-curl-7481-ae79e7" Nov 03 21:42:45.016689 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:45.016620 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-ae79e7" Nov 03 21:42:45.143085 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:45.143049 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb430705-431d-429a-bf48-e004d455e4c5-kube-api-access-2xt52\" (UniqueName: \"kubernetes.io/projected/fb430705-431d-429a-bf48-e004d455e4c5-kube-api-access-2xt52\") pod \"fb430705-431d-429a-bf48-e004d455e4c5\" (UID: \"fb430705-431d-429a-bf48-e004d455e4c5\") " Nov 03 21:42:45.145385 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:45.145346 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb430705-431d-429a-bf48-e004d455e4c5-kube-api-access-2xt52" pod "fb430705-431d-429a-bf48-e004d455e4c5" (UID: "fb430705-431d-429a-bf48-e004d455e4c5"). InnerVolumeSpecName "kube-api-access-2xt52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:42:45.244739 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:45.244706 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xt52\" (UniqueName: \"kubernetes.io/projected/fb430705-431d-429a-bf48-e004d455e4c5-kube-api-access-2xt52\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:42:45.776627 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:45.776585 4963 generic.go:356] "Generic (PLEG): container finished" podID="fb430705-431d-429a-bf48-e004d455e4c5" containerID="9ae0ee9f9cfdd30b0785a95df1d11c8f8ba438ad9f0dd40e64834d43db0141a7" exitCode=137 Nov 03 21:42:45.776627 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:45.776630 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-ae79e7" Nov 03 21:42:45.776870 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:45.776684 4963 scope.go:122] "RemoveContainer" containerID="9ae0ee9f9cfdd30b0785a95df1d11c8f8ba438ad9f0dd40e64834d43db0141a7" Nov 03 21:42:45.786505 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:45.786485 4963 scope.go:122] "RemoveContainer" containerID="9ae0ee9f9cfdd30b0785a95df1d11c8f8ba438ad9f0dd40e64834d43db0141a7" Nov 03 21:42:45.786809 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:42:45.786788 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae0ee9f9cfdd30b0785a95df1d11c8f8ba438ad9f0dd40e64834d43db0141a7\": container with ID starting with 9ae0ee9f9cfdd30b0785a95df1d11c8f8ba438ad9f0dd40e64834d43db0141a7 not found: ID does not exist" containerID="9ae0ee9f9cfdd30b0785a95df1d11c8f8ba438ad9f0dd40e64834d43db0141a7" Nov 03 21:42:45.786880 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:45.786825 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae0ee9f9cfdd30b0785a95df1d11c8f8ba438ad9f0dd40e64834d43db0141a7"} err="failed to get container status \"9ae0ee9f9cfdd30b0785a95df1d11c8f8ba438ad9f0dd40e64834d43db0141a7\": rpc error: code = NotFound desc = could not find container \"9ae0ee9f9cfdd30b0785a95df1d11c8f8ba438ad9f0dd40e64834d43db0141a7\": container with ID starting with 9ae0ee9f9cfdd30b0785a95df1d11c8f8ba438ad9f0dd40e64834d43db0141a7 not found: ID does not exist" Nov 03 21:42:45.834747 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:42:45.834699 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7481-ae79e7\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="fb430705-431d-429a-bf48-e004d455e4c5" pod="opendatahub/test-curl-7481-ae79e7" Nov 03 21:42:46.803452 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:46.803418 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:46.968266 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:46.968227 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb430705-431d-429a-bf48-e004d455e4c5" path="/var/lib/kubelet/pods/fb430705-431d-429a-bf48-e004d455e4c5/volumes" Nov 03 21:42:47.809750 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:47.809721 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:48.004563 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:48.004528 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:50.414793 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:50.414760 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:50.623599 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:50.623565 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:51.382241 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:51.382207 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:42:55.910697 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:42:55.910661 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:00.375694 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:00.375640 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:04.077273 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:04.077241 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:04.261487 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:04.261433 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:04.905465 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:04.905403 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:09.495534 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:09.495479 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:12.531528 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:43:12.531486 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:43:14.179403 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:14.179371 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:18.647450 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:18.647366 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:23.250146 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:23.250112 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:25.574143 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:25.574095 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:27.974815 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:27.974773 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:32.379053 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.379011 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6"] Nov 03 21:43:32.379820 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.379802 4963 cpu_manager.go:414] "containerMap: removing container" podUID="fb430705-431d-429a-bf48-e004d455e4c5" containerName="test-curl-7481-ae79e7" Nov 03 21:43:32.379866 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.379824 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb430705-431d-429a-bf48-e004d455e4c5" containerName="test-curl-7481-ae79e7" Nov 03 21:43:32.380056 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.380041 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb430705-431d-429a-bf48-e004d455e4c5" containerName="test-curl-7481-ae79e7" Nov 03 21:43:32.383804 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.383782 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:43:32.474332 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.474278 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34356d65-dc53-4d4c-bffe-341dbd8653b3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6\" (UID: \"34356d65-dc53-4d4c-bffe-341dbd8653b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:43:32.474544 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.474440 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh5v7\" (UniqueName: \"kubernetes.io/projected/34356d65-dc53-4d4c-bffe-341dbd8653b3-kube-api-access-bh5v7\") pod \"maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6\" (UID: \"34356d65-dc53-4d4c-bffe-341dbd8653b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:43:32.474544 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.474491 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34356d65-dc53-4d4c-bffe-341dbd8653b3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6\" (UID: \"34356d65-dc53-4d4c-bffe-341dbd8653b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:43:32.474702 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.474688 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34356d65-dc53-4d4c-bffe-341dbd8653b3-metrics-certs\") pod \"maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6\" (UID: \"34356d65-dc53-4d4c-bffe-341dbd8653b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:43:32.481677 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.481640 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-2057ca-serving-cert\"" Nov 03 21:43:32.484972 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.484943 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6"] Nov 03 21:43:32.575533 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.575495 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh5v7\" (UniqueName: \"kubernetes.io/projected/34356d65-dc53-4d4c-bffe-341dbd8653b3-kube-api-access-bh5v7\") pod \"maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6\" (UID: \"34356d65-dc53-4d4c-bffe-341dbd8653b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:43:32.575711 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.575552 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34356d65-dc53-4d4c-bffe-341dbd8653b3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6\" (UID: \"34356d65-dc53-4d4c-bffe-341dbd8653b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:43:32.575711 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.575628 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34356d65-dc53-4d4c-bffe-341dbd8653b3-metrics-certs\") pod \"maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6\" (UID: \"34356d65-dc53-4d4c-bffe-341dbd8653b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:43:32.575711 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.575654 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34356d65-dc53-4d4c-bffe-341dbd8653b3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6\" (UID: \"34356d65-dc53-4d4c-bffe-341dbd8653b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:43:32.576236 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:43:32.575715 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-serving-cert: secret "maas-api-e2e-extmod-a-2057ca-serving-cert" not found Nov 03 21:43:32.576236 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:43:32.575785 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34356d65-dc53-4d4c-bffe-341dbd8653b3-maas-api-tls podName:34356d65-dc53-4d4c-bffe-341dbd8653b3 nodeName:}" failed. No retries permitted until 2026-11-03 21:43:33.075764658 +0000 UTC m=+3362.235836908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34356d65-dc53-4d4c-bffe-341dbd8653b3-maas-api-tls") pod "maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" (UID: "34356d65-dc53-4d4c-bffe-341dbd8653b3") : secret "maas-api-e2e-extmod-a-2057ca-serving-cert" not found Nov 03 21:43:32.576374 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.576357 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34356d65-dc53-4d4c-bffe-341dbd8653b3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6\" (UID: \"34356d65-dc53-4d4c-bffe-341dbd8653b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:43:32.578149 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.578133 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34356d65-dc53-4d4c-bffe-341dbd8653b3-metrics-certs\") pod \"maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6\" (UID: \"34356d65-dc53-4d4c-bffe-341dbd8653b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:43:32.602622 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.602586 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:32.635962 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.635850 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh5v7\" (UniqueName: \"kubernetes.io/projected/34356d65-dc53-4d4c-bffe-341dbd8653b3-kube-api-access-bh5v7\") pod \"maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6\" (UID: \"34356d65-dc53-4d4c-bffe-341dbd8653b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:43:32.807896 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:32.807862 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:33.033797 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:33.033766 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:33.082076 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:33.082041 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34356d65-dc53-4d4c-bffe-341dbd8653b3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6\" (UID: \"34356d65-dc53-4d4c-bffe-341dbd8653b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:43:33.084523 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:33.084498 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34356d65-dc53-4d4c-bffe-341dbd8653b3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6\" (UID: \"34356d65-dc53-4d4c-bffe-341dbd8653b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:43:33.297798 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:33.297715 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:43:33.675558 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:33.675477 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6"] Nov 03 21:43:33.681717 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:43:33.681691 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34356d65_dc53_4d4c_bffe_341dbd8653b3.slice/crio-c07c0162f9128a75a824cd802dd6da45859fcc6b11f6eb496dc6d4ee25c1857d WatchSource:0}: Error finding container c07c0162f9128a75a824cd802dd6da45859fcc6b11f6eb496dc6d4ee25c1857d: Status 404 returned error can't find the container with id c07c0162f9128a75a824cd802dd6da45859fcc6b11f6eb496dc6d4ee25c1857d Nov 03 21:43:34.081632 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:34.081595 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" event={"ID":"34356d65-dc53-4d4c-bffe-341dbd8653b3","Type":"ContainerStarted","Data":"c07c0162f9128a75a824cd802dd6da45859fcc6b11f6eb496dc6d4ee25c1857d"} Nov 03 21:43:35.097615 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:35.097570 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" event={"ID":"34356d65-dc53-4d4c-bffe-341dbd8653b3","Type":"ContainerStarted","Data":"611b51a8c4a588430ac0719d50c0ac9929f9fb786a157dea9cf44caed8b0d0b7"} Nov 03 21:43:35.098053 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:35.097670 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:43:35.220294 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:35.220239 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" podStartSLOduration=2.348103763 podStartE2EDuration="3.220223676s" totalImagesPullingTime="872.119913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:43:35.218375386 +0000 UTC m=+3364.378447659" watchObservedRunningTime="2026-11-03 21:43:35.220223676 +0000 UTC m=+3364.380295950" Nov 03 21:43:37.589819 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:37.589781 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:38.097360 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:38.097326 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:38.836346 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:38.836311 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:39.046451 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:39.046415 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:39.984644 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:39.984563 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:40.504256 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:40.504215 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz"] Nov 03 21:43:40.581622 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:40.581588 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:41.111048 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:41.111020 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:43:41.217551 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:41.217515 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:41.883938 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:41.883894 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:42.072414 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:42.072384 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:42.324214 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:42.324176 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:42.529215 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:43:42.529171 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:43:43.491080 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:43.491049 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:43.701591 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:43.701519 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:45.585232 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:45.585198 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:47.501439 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:47.501405 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:47.683261 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:47.683226 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:50.070235 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:50.070195 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:51.608138 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:51.608078 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:51.795211 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:51.795179 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:51.920999 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:51.920919 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss"] Nov 03 21:43:51.925263 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:51.925241 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:43:51.977322 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:51.977287 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-d5e2db-serving-cert\"" Nov 03 21:43:52.006060 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.006025 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16279a5e-a28a-44a6-ba3c-05566f417084-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss\" (UID: \"16279a5e-a28a-44a6-ba3c-05566f417084\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:43:52.006251 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.006078 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16279a5e-a28a-44a6-ba3c-05566f417084-metrics-certs\") pod \"maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss\" (UID: \"16279a5e-a28a-44a6-ba3c-05566f417084\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:43:52.006251 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.006202 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2g8c\" (UniqueName: \"kubernetes.io/projected/16279a5e-a28a-44a6-ba3c-05566f417084-kube-api-access-s2g8c\") pod \"maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss\" (UID: \"16279a5e-a28a-44a6-ba3c-05566f417084\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:43:52.006251 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.006241 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16279a5e-a28a-44a6-ba3c-05566f417084-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss\" (UID: \"16279a5e-a28a-44a6-ba3c-05566f417084\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:43:52.027973 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.027943 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss"] Nov 03 21:43:52.030486 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.030462 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j"] Nov 03 21:43:52.034397 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.034378 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j" Nov 03 21:43:52.088628 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.088586 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j"] Nov 03 21:43:52.089892 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.089869 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-a02404-dockercfg-vncm9\"" Nov 03 21:43:52.090193 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.090179 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-a02404\"" Nov 03 21:43:52.107714 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.107679 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16279a5e-a28a-44a6-ba3c-05566f417084-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss\" (UID: \"16279a5e-a28a-44a6-ba3c-05566f417084\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:43:52.107954 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.107741 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16279a5e-a28a-44a6-ba3c-05566f417084-metrics-certs\") pod \"maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss\" (UID: \"16279a5e-a28a-44a6-ba3c-05566f417084\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:43:52.107954 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:43:52.107803 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-serving-cert: secret "maas-api-e2e-extmod-b-d5e2db-serving-cert" not found Nov 03 21:43:52.107954 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.107835 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2g8c\" (UniqueName: \"kubernetes.io/projected/16279a5e-a28a-44a6-ba3c-05566f417084-kube-api-access-s2g8c\") pod \"maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss\" (UID: \"16279a5e-a28a-44a6-ba3c-05566f417084\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:43:52.107954 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:43:52.107879 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16279a5e-a28a-44a6-ba3c-05566f417084-maas-api-tls podName:16279a5e-a28a-44a6-ba3c-05566f417084 nodeName:}" failed. No retries permitted until 2026-11-03 21:43:52.607857133 +0000 UTC m=+3381.767929397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16279a5e-a28a-44a6-ba3c-05566f417084-maas-api-tls") pod "maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" (UID: "16279a5e-a28a-44a6-ba3c-05566f417084") : secret "maas-api-e2e-extmod-b-d5e2db-serving-cert" not found Nov 03 21:43:52.107954 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.107922 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6dx7\" (UniqueName: \"kubernetes.io/projected/eaebca48-d00b-4f5b-9a25-67f09d3c5f45-kube-api-access-r6dx7\") pod \"payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j\" (UID: \"eaebca48-d00b-4f5b-9a25-67f09d3c5f45\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j" Nov 03 21:43:52.108403 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.107983 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16279a5e-a28a-44a6-ba3c-05566f417084-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss\" (UID: \"16279a5e-a28a-44a6-ba3c-05566f417084\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:43:52.108403 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.108071 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaebca48-d00b-4f5b-9a25-67f09d3c5f45-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j\" (UID: \"eaebca48-d00b-4f5b-9a25-67f09d3c5f45\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j" Nov 03 21:43:52.109209 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.109185 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16279a5e-a28a-44a6-ba3c-05566f417084-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss\" (UID: \"16279a5e-a28a-44a6-ba3c-05566f417084\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:43:52.111402 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.111363 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16279a5e-a28a-44a6-ba3c-05566f417084-metrics-certs\") pod \"maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss\" (UID: \"16279a5e-a28a-44a6-ba3c-05566f417084\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:43:52.163460 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.163429 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:52.169708 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.169673 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2g8c\" (UniqueName: \"kubernetes.io/projected/16279a5e-a28a-44a6-ba3c-05566f417084-kube-api-access-s2g8c\") pod \"maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss\" (UID: \"16279a5e-a28a-44a6-ba3c-05566f417084\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:43:52.208870 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.208818 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaebca48-d00b-4f5b-9a25-67f09d3c5f45-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j\" (UID: \"eaebca48-d00b-4f5b-9a25-67f09d3c5f45\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j" Nov 03 21:43:52.209101 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.209057 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6dx7\" (UniqueName: \"kubernetes.io/projected/eaebca48-d00b-4f5b-9a25-67f09d3c5f45-kube-api-access-r6dx7\") pod \"payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j\" (UID: \"eaebca48-d00b-4f5b-9a25-67f09d3c5f45\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j" Nov 03 21:43:52.210805 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.210776 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaebca48-d00b-4f5b-9a25-67f09d3c5f45-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j\" (UID: \"eaebca48-d00b-4f5b-9a25-67f09d3c5f45\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j" Nov 03 21:43:52.268519 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.268486 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6dx7\" (UniqueName: \"kubernetes.io/projected/eaebca48-d00b-4f5b-9a25-67f09d3c5f45-kube-api-access-r6dx7\") pod \"payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j\" (UID: \"eaebca48-d00b-4f5b-9a25-67f09d3c5f45\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j" Nov 03 21:43:52.348107 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.348064 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j" Nov 03 21:43:52.353540 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.353501 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:52.534552 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.534443 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:52.614352 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.614313 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16279a5e-a28a-44a6-ba3c-05566f417084-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss\" (UID: \"16279a5e-a28a-44a6-ba3c-05566f417084\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:43:52.616759 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.616738 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16279a5e-a28a-44a6-ba3c-05566f417084-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss\" (UID: \"16279a5e-a28a-44a6-ba3c-05566f417084\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:43:52.725069 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.725024 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j"] Nov 03 21:43:52.730278 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:43:52.730249 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaebca48_d00b_4f5b_9a25_67f09d3c5f45.slice/crio-3b098c59a4107f46f0df5f921692a715b7240c4ab6dba322ac60e05314c84d78 WatchSource:0}: Error finding container 3b098c59a4107f46f0df5f921692a715b7240c4ab6dba322ac60e05314c84d78: Status 404 returned error can't find the container with id 3b098c59a4107f46f0df5f921692a715b7240c4ab6dba322ac60e05314c84d78 Nov 03 21:43:52.839467 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:52.839433 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:43:53.218886 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:53.218847 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss"] Nov 03 21:43:53.223211 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:53.223177 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j" event={"ID":"eaebca48-d00b-4f5b-9a25-67f09d3c5f45","Type":"ContainerStarted","Data":"5691e9bafa4a44569631755342f43c916e25458a42e1b5d13ba354c700739da1"} Nov 03 21:43:53.223321 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:53.223237 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j" event={"ID":"eaebca48-d00b-4f5b-9a25-67f09d3c5f45","Type":"ContainerStarted","Data":"3b098c59a4107f46f0df5f921692a715b7240c4ab6dba322ac60e05314c84d78"} Nov 03 21:43:53.223394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:53.223357 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j" Nov 03 21:43:53.223888 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:43:53.223866 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16279a5e_a28a_44a6_ba3c_05566f417084.slice/crio-22949dadc08d5286763d5e210799d49ca418a5eba050997178f0a4419f976dea WatchSource:0}: Error finding container 22949dadc08d5286763d5e210799d49ca418a5eba050997178f0a4419f976dea: Status 404 returned error can't find the container with id 22949dadc08d5286763d5e210799d49ca418a5eba050997178f0a4419f976dea Nov 03 21:43:53.338062 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:53.337967 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j" podStartSLOduration=2.337950355 podStartE2EDuration="2.337950355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:43:53.335273784 +0000 UTC m=+3382.495346055" watchObservedRunningTime="2026-11-03 21:43:53.337950355 +0000 UTC m=+3382.498022624" Nov 03 21:43:54.235406 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:54.235356 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" event={"ID":"16279a5e-a28a-44a6-ba3c-05566f417084","Type":"ContainerStarted","Data":"22949dadc08d5286763d5e210799d49ca418a5eba050997178f0a4419f976dea"} Nov 03 21:43:54.254515 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:54.254484 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:54.444095 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:54.444061 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:43:55.242818 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.242780 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" event={"ID":"16279a5e-a28a-44a6-ba3c-05566f417084","Type":"ContainerStarted","Data":"6d66892a6c407c099ff7037ce7932705283f4b4e399da260aa96ec879b707723"} Nov 03 21:43:55.243297 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.242846 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:43:55.356950 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.356880 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" podStartSLOduration=3.423891342 podStartE2EDuration="4.356866369s" totalImagesPullingTime="932.975027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:43:55.354433753 +0000 UTC m=+3384.514506026" watchObservedRunningTime="2026-11-03 21:43:55.356866369 +0000 UTC m=+3384.516938640" Nov 03 21:43:55.524863 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.524766 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" podUID="02a30a91-1bf3-422b-9f88-3023c4431e1d" containerName="main" containerID="cri-o://5b18349dde0652af894cb28dd9f388d8aa9b5593f847275ab567224ed0062439" gracePeriod=45 Nov 03 21:43:55.792046 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.791980 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:43:55.849746 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.849714 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a30a91-1bf3-422b-9f88-3023c4431e1d-kube-api-access-44g9j\" (UniqueName: \"kubernetes.io/projected/02a30a91-1bf3-422b-9f88-3023c4431e1d-kube-api-access-44g9j\") pod \"02a30a91-1bf3-422b-9f88-3023c4431e1d\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " Nov 03 21:43:55.849960 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.849783 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-tmp-dir\") pod \"02a30a91-1bf3-422b-9f88-3023c4431e1d\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " Nov 03 21:43:55.849960 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.849877 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-home\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-home\") pod \"02a30a91-1bf3-422b-9f88-3023c4431e1d\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " Nov 03 21:43:55.849960 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.849948 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-model-cache\") pod \"02a30a91-1bf3-422b-9f88-3023c4431e1d\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " Nov 03 21:43:55.850160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.849991 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-dshm\") pod \"02a30a91-1bf3-422b-9f88-3023c4431e1d\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " Nov 03 21:43:55.850160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.850038 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02a30a91-1bf3-422b-9f88-3023c4431e1d-tls-certs\" (UniqueName: \"kubernetes.io/secret/02a30a91-1bf3-422b-9f88-3023c4431e1d-tls-certs\") pod \"02a30a91-1bf3-422b-9f88-3023c4431e1d\" (UID: \"02a30a91-1bf3-422b-9f88-3023c4431e1d\") " Nov 03 21:43:55.850160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.850068 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-tmp-dir" pod "02a30a91-1bf3-422b-9f88-3023c4431e1d" (UID: "02a30a91-1bf3-422b-9f88-3023c4431e1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:55.850285 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.850151 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-home" pod "02a30a91-1bf3-422b-9f88-3023c4431e1d" (UID: "02a30a91-1bf3-422b-9f88-3023c4431e1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:55.850285 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.850215 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-model-cache" pod "02a30a91-1bf3-422b-9f88-3023c4431e1d" (UID: "02a30a91-1bf3-422b-9f88-3023c4431e1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:55.850606 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.850583 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:43:55.850606 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.850606 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:43:55.850753 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.850618 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:43:55.852137 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.852117 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a30a91-1bf3-422b-9f88-3023c4431e1d-kube-api-access-44g9j" pod "02a30a91-1bf3-422b-9f88-3023c4431e1d" (UID: "02a30a91-1bf3-422b-9f88-3023c4431e1d"). InnerVolumeSpecName "kube-api-access-44g9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:43:55.852215 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.852160 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-dshm" pod "02a30a91-1bf3-422b-9f88-3023c4431e1d" (UID: "02a30a91-1bf3-422b-9f88-3023c4431e1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:43:55.852260 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.852237 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a30a91-1bf3-422b-9f88-3023c4431e1d-tls-certs" pod "02a30a91-1bf3-422b-9f88-3023c4431e1d" (UID: "02a30a91-1bf3-422b-9f88-3023c4431e1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:43:55.952008 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.951972 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44g9j\" (UniqueName: \"kubernetes.io/projected/02a30a91-1bf3-422b-9f88-3023c4431e1d-kube-api-access-44g9j\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:43:55.952008 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.952000 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a30a91-1bf3-422b-9f88-3023c4431e1d-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:43:55.952008 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:55.952012 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a30a91-1bf3-422b-9f88-3023c4431e1d-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:43:56.250203 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:56.250170 4963 generic.go:356] "Generic (PLEG): container finished" podID="02a30a91-1bf3-422b-9f88-3023c4431e1d" containerID="5b18349dde0652af894cb28dd9f388d8aa9b5593f847275ab567224ed0062439" exitCode=0 Nov 03 21:43:56.250706 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:56.250217 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" Nov 03 21:43:56.250706 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:56.250253 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" event={"ID":"02a30a91-1bf3-422b-9f88-3023c4431e1d","Type":"ContainerDied","Data":"5b18349dde0652af894cb28dd9f388d8aa9b5593f847275ab567224ed0062439"} Nov 03 21:43:56.250706 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:56.250304 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz" event={"ID":"02a30a91-1bf3-422b-9f88-3023c4431e1d","Type":"ContainerDied","Data":"ba64281b21aec3eb895b597800f3cb6b0cc71e63bb753ed77bd2b947f6a0f9b5"} Nov 03 21:43:56.250706 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:56.250327 4963 scope.go:122] "RemoveContainer" containerID="5b18349dde0652af894cb28dd9f388d8aa9b5593f847275ab567224ed0062439" Nov 03 21:43:56.260644 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:56.260623 4963 scope.go:122] "RemoveContainer" containerID="5b18349dde0652af894cb28dd9f388d8aa9b5593f847275ab567224ed0062439" Nov 03 21:43:56.260941 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:43:56.260898 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b18349dde0652af894cb28dd9f388d8aa9b5593f847275ab567224ed0062439\": container with ID starting with 5b18349dde0652af894cb28dd9f388d8aa9b5593f847275ab567224ed0062439 not found: ID does not exist" containerID="5b18349dde0652af894cb28dd9f388d8aa9b5593f847275ab567224ed0062439" Nov 03 21:43:56.260994 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:56.260951 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b18349dde0652af894cb28dd9f388d8aa9b5593f847275ab567224ed0062439"} err="failed to get container status \"5b18349dde0652af894cb28dd9f388d8aa9b5593f847275ab567224ed0062439\": rpc error: code = NotFound desc = could not find container \"5b18349dde0652af894cb28dd9f388d8aa9b5593f847275ab567224ed0062439\": container with ID starting with 5b18349dde0652af894cb28dd9f388d8aa9b5593f847275ab567224ed0062439 not found: ID does not exist" Nov 03 21:43:56.407472 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:56.407408 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz"] Nov 03 21:43:56.412438 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:56.412392 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-714979/auth-test-model-714979-kserve-6bd4f49845-tk9xz"] Nov 03 21:43:56.968032 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:56.967999 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02a30a91-1bf3-422b-9f88-3023c4431e1d" path="/var/lib/kubelet/pods/02a30a91-1bf3-422b-9f88-3023c4431e1d/volumes" Nov 03 21:43:57.330451 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:43:57.330353 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:01.256838 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:01.256808 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:44:01.996510 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:01.996472 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:04.239472 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:04.239442 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j" Nov 03 21:44:06.178246 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.178209 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf"] Nov 03 21:44:06.178799 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.178784 4963 cpu_manager.go:414] "containerMap: removing container" podUID="02a30a91-1bf3-422b-9f88-3023c4431e1d" containerName="main" Nov 03 21:44:06.178843 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.178801 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a30a91-1bf3-422b-9f88-3023c4431e1d" containerName="main" Nov 03 21:44:06.178952 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.178941 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a30a91-1bf3-422b-9f88-3023c4431e1d" containerName="main" Nov 03 21:44:06.182468 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.182449 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" Nov 03 21:44:06.233591 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.233540 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-c0fc5f-serving-cert\"" Nov 03 21:44:06.283650 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.283615 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf"] Nov 03 21:44:06.376842 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.376806 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-metrics-certs\") pod \"maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf\" (UID: \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" Nov 03 21:44:06.377076 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.376853 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf\" (UID: \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" Nov 03 21:44:06.377076 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.376964 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf\" (UID: \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" Nov 03 21:44:06.377196 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.377084 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl8kk\" (UniqueName: \"kubernetes.io/projected/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-kube-api-access-vl8kk\") pod \"maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf\" (UID: \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" Nov 03 21:44:06.399418 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.399381 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:06.478754 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.478713 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl8kk\" (UniqueName: \"kubernetes.io/projected/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-kube-api-access-vl8kk\") pod \"maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf\" (UID: \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" Nov 03 21:44:06.479014 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.478823 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-metrics-certs\") pod \"maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf\" (UID: \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" Nov 03 21:44:06.479014 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.478856 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf\" (UID: \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" Nov 03 21:44:06.479014 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.478894 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf\" (UID: \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" Nov 03 21:44:06.479663 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.479641 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf\" (UID: \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" Nov 03 21:44:06.481440 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.481411 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf\" (UID: \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" Nov 03 21:44:06.481586 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.481564 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-metrics-certs\") pod \"maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf\" (UID: \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" Nov 03 21:44:06.537717 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.537683 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl8kk\" (UniqueName: \"kubernetes.io/projected/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-kube-api-access-vl8kk\") pod \"maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf\" (UID: \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" Nov 03 21:44:06.589386 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.589350 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:06.769155 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.769065 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:06.796423 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:06.796379 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" Nov 03 21:44:07.175053 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:07.174962 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf"] Nov 03 21:44:07.180048 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:44:07.180017 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1f5ae3_b769_4b92_8f54_f45c2393a4e7.slice/crio-a3b3f8ceaae1820f9c0dde9de520405ebbb0944fdf1d46c8614096609e78eacd WatchSource:0}: Error finding container a3b3f8ceaae1820f9c0dde9de520405ebbb0944fdf1d46c8614096609e78eacd: Status 404 returned error can't find the container with id a3b3f8ceaae1820f9c0dde9de520405ebbb0944fdf1d46c8614096609e78eacd Nov 03 21:44:07.317451 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:07.317414 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" event={"ID":"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7","Type":"ContainerStarted","Data":"a3b3f8ceaae1820f9c0dde9de520405ebbb0944fdf1d46c8614096609e78eacd"} Nov 03 21:44:08.326776 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:08.326669 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" event={"ID":"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7","Type":"ContainerStarted","Data":"2e0f301273b8a2b633d82e7265b5638a0081ee11ae63ef00adc57d5e59f08779"} Nov 03 21:44:08.327203 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:08.326935 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" Nov 03 21:44:08.441026 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:08.440970 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" podStartSLOduration=1.548456602 podStartE2EDuration="2.440954678s" totalImagesPullingTime="892.498076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:08.438706804 +0000 UTC m=+3397.598779075" watchObservedRunningTime="2026-11-03 21:44:08.440954678 +0000 UTC m=+3397.601026950" Nov 03 21:44:11.166014 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:11.165979 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:12.531383 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:44:12.531335 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:44:14.338457 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:14.338425 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" Nov 03 21:44:15.711752 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:15.711718 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:20.195575 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:20.195539 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:24.653975 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:24.653931 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:29.212218 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:29.212166 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:33.723860 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:33.723823 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:38.247632 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:38.247579 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:40.301931 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:40.301873 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:42.528545 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:44:42.528508 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:44:42.747874 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:42.747838 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:47.325346 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:47.325314 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:51.843184 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:51.843150 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:52.903888 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:52.903854 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:44:56.279703 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:44:56.279669 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:00.189801 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.189765 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89"] Nov 03 21:45:00.195370 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.195345 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" Nov 03 21:45:00.210230 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.210201 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5"] Nov 03 21:45:00.214138 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.214114 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" Nov 03 21:45:00.216811 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.216785 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p"] Nov 03 21:45:00.220634 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.220615 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" Nov 03 21:45:00.237702 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.237678 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9"] Nov 03 21:45:00.241530 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.241506 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm"] Nov 03 21:45:00.241645 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.241598 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" Nov 03 21:45:00.246945 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.246890 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" Nov 03 21:45:00.301001 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.300952 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24xnx\" (UniqueName: \"kubernetes.io/projected/02bd2073-42ed-4acd-94b6-f3002d558c5a-kube-api-access-24xnx\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89\" (UID: \"02bd2073-42ed-4acd-94b6-f3002d558c5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" Nov 03 21:45:00.301195 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.301013 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02bd2073-42ed-4acd-94b6-f3002d558c5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89\" (UID: \"02bd2073-42ed-4acd-94b6-f3002d558c5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" Nov 03 21:45:00.312086 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.312047 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89"] Nov 03 21:45:00.323773 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.323739 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\"" Nov 03 21:45:00.355312 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.355274 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p"] Nov 03 21:45:00.359162 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.359126 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5"] Nov 03 21:45:00.361980 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.361952 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm"] Nov 03 21:45:00.385376 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.385335 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9"] Nov 03 21:45:00.402022 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.401983 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x56k\" (UniqueName: \"kubernetes.io/projected/4f568d8c-8808-4c5c-b862-fcdc67e8aa7c-kube-api-access-9x56k\") pod \"maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm\" (UID: \"4f568d8c-8808-4c5c-b862-fcdc67e8aa7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" Nov 03 21:45:00.402196 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.402115 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrps5\" (UniqueName: \"kubernetes.io/projected/27132aa5-cb8f-45a4-9ca8-2998ba4f70dc-kube-api-access-lrps5\") pod \"maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9\" (UID: \"27132aa5-cb8f-45a4-9ca8-2998ba4f70dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" Nov 03 21:45:00.402292 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.402271 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98347d8-c132-4714-98fb-4642418cec20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5\" (UID: \"d98347d8-c132-4714-98fb-4642418cec20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" Nov 03 21:45:00.402327 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.402313 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhk6h\" (UniqueName: \"kubernetes.io/projected/d98347d8-c132-4714-98fb-4642418cec20-kube-api-access-xhk6h\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5\" (UID: \"d98347d8-c132-4714-98fb-4642418cec20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" Nov 03 21:45:00.402457 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.402430 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f568d8c-8808-4c5c-b862-fcdc67e8aa7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm\" (UID: \"4f568d8c-8808-4c5c-b862-fcdc67e8aa7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" Nov 03 21:45:00.402531 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.402509 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24xnx\" (UniqueName: \"kubernetes.io/projected/02bd2073-42ed-4acd-94b6-f3002d558c5a-kube-api-access-24xnx\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89\" (UID: \"02bd2073-42ed-4acd-94b6-f3002d558c5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" Nov 03 21:45:00.402569 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.402547 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91983bf-8cf7-4259-99f2-1139c1c9a5c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p\" (UID: \"c91983bf-8cf7-4259-99f2-1139c1c9a5c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" Nov 03 21:45:00.402606 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.402567 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02bd2073-42ed-4acd-94b6-f3002d558c5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89\" (UID: \"02bd2073-42ed-4acd-94b6-f3002d558c5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" Nov 03 21:45:00.402606 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.402586 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27132aa5-cb8f-45a4-9ca8-2998ba4f70dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9\" (UID: \"27132aa5-cb8f-45a4-9ca8-2998ba4f70dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" Nov 03 21:45:00.402606 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.402602 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrnmd\" (UniqueName: \"kubernetes.io/projected/c91983bf-8cf7-4259-99f2-1139c1c9a5c1-kube-api-access-qrnmd\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p\" (UID: \"c91983bf-8cf7-4259-99f2-1139c1c9a5c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" Nov 03 21:45:00.403292 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.403266 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02bd2073-42ed-4acd-94b6-f3002d558c5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89\" (UID: \"02bd2073-42ed-4acd-94b6-f3002d558c5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" Nov 03 21:45:00.461336 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.461299 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24xnx\" (UniqueName: \"kubernetes.io/projected/02bd2073-42ed-4acd-94b6-f3002d558c5a-kube-api-access-24xnx\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89\" (UID: \"02bd2073-42ed-4acd-94b6-f3002d558c5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" Nov 03 21:45:00.503932 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.503868 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27132aa5-cb8f-45a4-9ca8-2998ba4f70dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9\" (UID: \"27132aa5-cb8f-45a4-9ca8-2998ba4f70dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" Nov 03 21:45:00.504116 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.503939 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrnmd\" (UniqueName: \"kubernetes.io/projected/c91983bf-8cf7-4259-99f2-1139c1c9a5c1-kube-api-access-qrnmd\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p\" (UID: \"c91983bf-8cf7-4259-99f2-1139c1c9a5c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" Nov 03 21:45:00.504116 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.504005 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x56k\" (UniqueName: \"kubernetes.io/projected/4f568d8c-8808-4c5c-b862-fcdc67e8aa7c-kube-api-access-9x56k\") pod \"maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm\" (UID: \"4f568d8c-8808-4c5c-b862-fcdc67e8aa7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" Nov 03 21:45:00.504116 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.504079 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrps5\" (UniqueName: \"kubernetes.io/projected/27132aa5-cb8f-45a4-9ca8-2998ba4f70dc-kube-api-access-lrps5\") pod \"maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9\" (UID: \"27132aa5-cb8f-45a4-9ca8-2998ba4f70dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" Nov 03 21:45:00.504278 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.504183 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98347d8-c132-4714-98fb-4642418cec20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5\" (UID: \"d98347d8-c132-4714-98fb-4642418cec20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" Nov 03 21:45:00.504278 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.504226 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhk6h\" (UniqueName: \"kubernetes.io/projected/d98347d8-c132-4714-98fb-4642418cec20-kube-api-access-xhk6h\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5\" (UID: \"d98347d8-c132-4714-98fb-4642418cec20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" Nov 03 21:45:00.504389 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.504282 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f568d8c-8808-4c5c-b862-fcdc67e8aa7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm\" (UID: \"4f568d8c-8808-4c5c-b862-fcdc67e8aa7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" Nov 03 21:45:00.504389 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.504328 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91983bf-8cf7-4259-99f2-1139c1c9a5c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p\" (UID: \"c91983bf-8cf7-4259-99f2-1139c1c9a5c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" Nov 03 21:45:00.504688 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.504660 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27132aa5-cb8f-45a4-9ca8-2998ba4f70dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9\" (UID: \"27132aa5-cb8f-45a4-9ca8-2998ba4f70dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" Nov 03 21:45:00.504833 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.504813 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98347d8-c132-4714-98fb-4642418cec20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5\" (UID: \"d98347d8-c132-4714-98fb-4642418cec20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" Nov 03 21:45:00.504954 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.504938 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f568d8c-8808-4c5c-b862-fcdc67e8aa7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm\" (UID: \"4f568d8c-8808-4c5c-b862-fcdc67e8aa7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" Nov 03 21:45:00.505043 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.505024 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91983bf-8cf7-4259-99f2-1139c1c9a5c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p\" (UID: \"c91983bf-8cf7-4259-99f2-1139c1c9a5c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" Nov 03 21:45:00.510633 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.510616 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" Nov 03 21:45:00.562533 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.562487 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x56k\" (UniqueName: \"kubernetes.io/projected/4f568d8c-8808-4c5c-b862-fcdc67e8aa7c-kube-api-access-9x56k\") pod \"maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm\" (UID: \"4f568d8c-8808-4c5c-b862-fcdc67e8aa7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" Nov 03 21:45:00.562735 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.562555 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrps5\" (UniqueName: \"kubernetes.io/projected/27132aa5-cb8f-45a4-9ca8-2998ba4f70dc-kube-api-access-lrps5\") pod \"maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9\" (UID: \"27132aa5-cb8f-45a4-9ca8-2998ba4f70dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" Nov 03 21:45:00.563007 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.562989 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhk6h\" (UniqueName: \"kubernetes.io/projected/d98347d8-c132-4714-98fb-4642418cec20-kube-api-access-xhk6h\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5\" (UID: \"d98347d8-c132-4714-98fb-4642418cec20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" Nov 03 21:45:00.563194 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.563171 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrnmd\" (UniqueName: \"kubernetes.io/projected/c91983bf-8cf7-4259-99f2-1139c1c9a5c1-kube-api-access-qrnmd\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p\" (UID: \"c91983bf-8cf7-4259-99f2-1139c1c9a5c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" Nov 03 21:45:00.564009 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.563753 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" Nov 03 21:45:00.761217 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.761120 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:00.827802 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.827769 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" Nov 03 21:45:00.835507 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.835485 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" Nov 03 21:45:00.857886 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.857859 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" Nov 03 21:45:00.892304 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.892186 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89"] Nov 03 21:45:00.992726 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:00.992701 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm"] Nov 03 21:45:01.214101 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:01.214064 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5"] Nov 03 21:45:01.219004 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:45:01.218978 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98347d8_c132_4714_98fb_4642418cec20.slice/crio-831c0d19d3f486281849d44be88d6b3a9795e50790ab3342a1d0882f01089b13 WatchSource:0}: Error finding container 831c0d19d3f486281849d44be88d6b3a9795e50790ab3342a1d0882f01089b13: Status 404 returned error can't find the container with id 831c0d19d3f486281849d44be88d6b3a9795e50790ab3342a1d0882f01089b13 Nov 03 21:45:01.264602 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:01.264566 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p"] Nov 03 21:45:01.269314 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:45:01.269283 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91983bf_8cf7_4259_99f2_1139c1c9a5c1.slice/crio-4a644ca9a82a4634d8dda4d25995613037697ab7f87e2fc1f57df63e10958ec5 WatchSource:0}: Error finding container 4a644ca9a82a4634d8dda4d25995613037697ab7f87e2fc1f57df63e10958ec5: Status 404 returned error can't find the container with id 4a644ca9a82a4634d8dda4d25995613037697ab7f87e2fc1f57df63e10958ec5 Nov 03 21:45:01.280349 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:01.280313 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9"] Nov 03 21:45:01.662536 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:01.662498 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" event={"ID":"d98347d8-c132-4714-98fb-4642418cec20","Type":"ContainerStarted","Data":"738d1f3a88d52f4fc7bd523d44c51a842e90c99427212b378127e9de8a0683ed"} Nov 03 21:45:01.662742 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:01.662561 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" event={"ID":"d98347d8-c132-4714-98fb-4642418cec20","Type":"ContainerStarted","Data":"831c0d19d3f486281849d44be88d6b3a9795e50790ab3342a1d0882f01089b13"} Nov 03 21:45:01.663814 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:01.663784 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" event={"ID":"27132aa5-cb8f-45a4-9ca8-2998ba4f70dc","Type":"ContainerStarted","Data":"143f1bba6f0999869fa2bc642019e737a904c0d14fcc6d743e847dd20d663b0d"} Nov 03 21:45:01.665385 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:01.665360 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" event={"ID":"02bd2073-42ed-4acd-94b6-f3002d558c5a","Type":"ContainerStarted","Data":"59ff283d4817167392cdf24401c23d2860f6574d4acc0cb00d54df75f2a1dd65"} Nov 03 21:45:01.665486 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:01.665411 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" event={"ID":"02bd2073-42ed-4acd-94b6-f3002d558c5a","Type":"ContainerStarted","Data":"3f891d9ba4c4f5618b815f5d3df2f13c6f778609e39ea9021c03162a8c6d8032"} Nov 03 21:45:01.666639 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:01.666606 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" event={"ID":"c91983bf-8cf7-4259-99f2-1139c1c9a5c1","Type":"ContainerStarted","Data":"4a644ca9a82a4634d8dda4d25995613037697ab7f87e2fc1f57df63e10958ec5"} Nov 03 21:45:01.668052 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:01.668028 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" event={"ID":"4f568d8c-8808-4c5c-b862-fcdc67e8aa7c","Type":"ContainerStarted","Data":"69cc8b578c2a183aa4be46af1eabf05995fbdd4c61a27919e8a3bac4e43f762c"} Nov 03 21:45:01.668143 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:01.668073 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" event={"ID":"4f568d8c-8808-4c5c-b862-fcdc67e8aa7c","Type":"ContainerStarted","Data":"d3c32f9246f803fcc356aaa7cadf21476b6bf4efc6cfa1ef0c038a07ae2ef8ad"} Nov 03 21:45:01.772956 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:01.772883 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" podStartSLOduration=1.772867561 podStartE2EDuration="1.772867561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:01.771076151 +0000 UTC m=+3450.931148425" watchObservedRunningTime="2026-11-03 21:45:01.772867561 +0000 UTC m=+3450.932939833" Nov 03 21:45:01.882722 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:01.882597 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" podStartSLOduration=1.882582054 podStartE2EDuration="1.882582054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:01.880391079 +0000 UTC m=+3451.040463350" watchObservedRunningTime="2026-11-03 21:45:01.882582054 +0000 UTC m=+3451.042654323" Nov 03 21:45:01.991773 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:01.991712 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" podStartSLOduration=1.991691838 podStartE2EDuration="1.991691838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:01.989362857 +0000 UTC m=+3451.149435130" watchObservedRunningTime="2026-11-03 21:45:01.991691838 +0000 UTC m=+3451.151764110" Nov 03 21:45:02.676134 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:02.676093 4963 generic.go:356] "Generic (PLEG): container finished" podID="c91983bf-8cf7-4259-99f2-1139c1c9a5c1" containerID="e6635a1c9f8298001b7c1292ef5c296379376f8620c4835f0b994beb5598524b" exitCode=60 Nov 03 21:45:02.676585 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:02.676182 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" event={"ID":"c91983bf-8cf7-4259-99f2-1139c1c9a5c1","Type":"ContainerDied","Data":"e6635a1c9f8298001b7c1292ef5c296379376f8620c4835f0b994beb5598524b"} Nov 03 21:45:02.676585 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:02.676459 4963 scope.go:122] "RemoveContainer" containerID="e6635a1c9f8298001b7c1292ef5c296379376f8620c4835f0b994beb5598524b" Nov 03 21:45:02.678100 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:02.678075 4963 generic.go:356] "Generic (PLEG): container finished" podID="4f568d8c-8808-4c5c-b862-fcdc67e8aa7c" containerID="69cc8b578c2a183aa4be46af1eabf05995fbdd4c61a27919e8a3bac4e43f762c" exitCode=60 Nov 03 21:45:02.678240 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:02.678186 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" event={"ID":"4f568d8c-8808-4c5c-b862-fcdc67e8aa7c","Type":"ContainerDied","Data":"69cc8b578c2a183aa4be46af1eabf05995fbdd4c61a27919e8a3bac4e43f762c"} Nov 03 21:45:02.678806 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:02.678786 4963 scope.go:122] "RemoveContainer" containerID="69cc8b578c2a183aa4be46af1eabf05995fbdd4c61a27919e8a3bac4e43f762c" Nov 03 21:45:02.680006 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:02.679984 4963 generic.go:356] "Generic (PLEG): container finished" podID="d98347d8-c132-4714-98fb-4642418cec20" containerID="738d1f3a88d52f4fc7bd523d44c51a842e90c99427212b378127e9de8a0683ed" exitCode=60 Nov 03 21:45:02.680103 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:02.680072 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" event={"ID":"d98347d8-c132-4714-98fb-4642418cec20","Type":"ContainerDied","Data":"738d1f3a88d52f4fc7bd523d44c51a842e90c99427212b378127e9de8a0683ed"} Nov 03 21:45:02.680340 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:02.680319 4963 scope.go:122] "RemoveContainer" containerID="738d1f3a88d52f4fc7bd523d44c51a842e90c99427212b378127e9de8a0683ed" Nov 03 21:45:02.681971 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:02.681845 4963 generic.go:356] "Generic (PLEG): container finished" podID="27132aa5-cb8f-45a4-9ca8-2998ba4f70dc" containerID="383d686bb35345f352e4439061baf6788598a52c91169085e1926ff8edc6e419" exitCode=60 Nov 03 21:45:02.681971 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:02.681935 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" event={"ID":"27132aa5-cb8f-45a4-9ca8-2998ba4f70dc","Type":"ContainerDied","Data":"383d686bb35345f352e4439061baf6788598a52c91169085e1926ff8edc6e419"} Nov 03 21:45:02.682200 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:02.682183 4963 scope.go:122] "RemoveContainer" containerID="383d686bb35345f352e4439061baf6788598a52c91169085e1926ff8edc6e419" Nov 03 21:45:02.683621 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:02.683600 4963 generic.go:356] "Generic (PLEG): container finished" podID="02bd2073-42ed-4acd-94b6-f3002d558c5a" containerID="59ff283d4817167392cdf24401c23d2860f6574d4acc0cb00d54df75f2a1dd65" exitCode=60 Nov 03 21:45:02.683706 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:02.683634 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" event={"ID":"02bd2073-42ed-4acd-94b6-f3002d558c5a","Type":"ContainerDied","Data":"59ff283d4817167392cdf24401c23d2860f6574d4acc0cb00d54df75f2a1dd65"} Nov 03 21:45:02.683847 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:02.683834 4963 scope.go:122] "RemoveContainer" containerID="59ff283d4817167392cdf24401c23d2860f6574d4acc0cb00d54df75f2a1dd65" Nov 03 21:45:03.694692 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:03.694654 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" event={"ID":"4f568d8c-8808-4c5c-b862-fcdc67e8aa7c","Type":"ContainerStarted","Data":"5018661a9ab8bca6783894a431079f8a40515c328401875ae6195e4390d34b88"} Nov 03 21:45:03.696887 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:03.696858 4963 generic.go:356] "Generic (PLEG): container finished" podID="d98347d8-c132-4714-98fb-4642418cec20" containerID="ff875bd39efef49844db07dd43ccaf3e5a0c93bf3ccf1d517d949776b9425d9e" exitCode=60 Nov 03 21:45:03.697034 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:03.696954 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" event={"ID":"d98347d8-c132-4714-98fb-4642418cec20","Type":"ContainerDied","Data":"ff875bd39efef49844db07dd43ccaf3e5a0c93bf3ccf1d517d949776b9425d9e"} Nov 03 21:45:03.697034 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:03.697011 4963 scope.go:122] "RemoveContainer" containerID="738d1f3a88d52f4fc7bd523d44c51a842e90c99427212b378127e9de8a0683ed" Nov 03 21:45:03.697262 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:03.697247 4963 scope.go:122] "RemoveContainer" containerID="ff875bd39efef49844db07dd43ccaf3e5a0c93bf3ccf1d517d949776b9425d9e" Nov 03 21:45:03.697598 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:03.697573 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5_odh-ai-gateway-infra(d98347d8-c132-4714-98fb-4642418cec20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" podUID="d98347d8-c132-4714-98fb-4642418cec20" Nov 03 21:45:03.699244 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:03.699211 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" event={"ID":"27132aa5-cb8f-45a4-9ca8-2998ba4f70dc","Type":"ContainerStarted","Data":"03ce6ef0cd3a2bb41d565705611827f4762ae96c2004914144b2cc52bf2e34d3"} Nov 03 21:45:03.701343 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:03.701310 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" event={"ID":"02bd2073-42ed-4acd-94b6-f3002d558c5a","Type":"ContainerStarted","Data":"c1955ab0ef3537793091405472ca85da9869c3e5b6b9f4d9134b8ef974428b5a"} Nov 03 21:45:03.701567 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:03.701547 4963 scope.go:122] "RemoveContainer" containerID="c1955ab0ef3537793091405472ca85da9869c3e5b6b9f4d9134b8ef974428b5a" Nov 03 21:45:03.701863 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:03.701839 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89_odh-ai-gateway-infra(02bd2073-42ed-4acd-94b6-f3002d558c5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" podUID="02bd2073-42ed-4acd-94b6-f3002d558c5a" Nov 03 21:45:03.703998 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:03.703976 4963 generic.go:356] "Generic (PLEG): container finished" podID="c91983bf-8cf7-4259-99f2-1139c1c9a5c1" containerID="1f350c030f52050ae6bdc99dce68319754917f9ed7eb078fa494ddcdf800b68b" exitCode=60 Nov 03 21:45:03.704125 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:03.704064 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" event={"ID":"c91983bf-8cf7-4259-99f2-1139c1c9a5c1","Type":"ContainerDied","Data":"1f350c030f52050ae6bdc99dce68319754917f9ed7eb078fa494ddcdf800b68b"} Nov 03 21:45:03.704386 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:03.704369 4963 scope.go:122] "RemoveContainer" containerID="1f350c030f52050ae6bdc99dce68319754917f9ed7eb078fa494ddcdf800b68b" Nov 03 21:45:03.704708 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:03.704682 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p_odh-ai-gateway-infra(c91983bf-8cf7-4259-99f2-1139c1c9a5c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" podUID="c91983bf-8cf7-4259-99f2-1139c1c9a5c1" Nov 03 21:45:03.963140 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:03.963114 4963 scope.go:122] "RemoveContainer" containerID="e6635a1c9f8298001b7c1292ef5c296379376f8620c4835f0b994beb5598524b" Nov 03 21:45:04.023400 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:04.023345 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" podStartSLOduration=4.023331481 podStartE2EDuration="4.023331481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:04.022263951 +0000 UTC m=+3453.182336253" watchObservedRunningTime="2026-11-03 21:45:04.023331481 +0000 UTC m=+3453.183403753" Nov 03 21:45:04.713944 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:04.713891 4963 scope.go:122] "RemoveContainer" containerID="1f350c030f52050ae6bdc99dce68319754917f9ed7eb078fa494ddcdf800b68b" Nov 03 21:45:04.714402 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:04.714242 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p_odh-ai-gateway-infra(c91983bf-8cf7-4259-99f2-1139c1c9a5c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" podUID="c91983bf-8cf7-4259-99f2-1139c1c9a5c1" Nov 03 21:45:04.715619 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:04.715592 4963 generic.go:356] "Generic (PLEG): container finished" podID="4f568d8c-8808-4c5c-b862-fcdc67e8aa7c" containerID="5018661a9ab8bca6783894a431079f8a40515c328401875ae6195e4390d34b88" exitCode=60 Nov 03 21:45:04.715752 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:04.715642 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" event={"ID":"4f568d8c-8808-4c5c-b862-fcdc67e8aa7c","Type":"ContainerDied","Data":"5018661a9ab8bca6783894a431079f8a40515c328401875ae6195e4390d34b88"} Nov 03 21:45:04.715752 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:04.715702 4963 scope.go:122] "RemoveContainer" containerID="69cc8b578c2a183aa4be46af1eabf05995fbdd4c61a27919e8a3bac4e43f762c" Nov 03 21:45:04.715993 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:04.715973 4963 scope.go:122] "RemoveContainer" containerID="5018661a9ab8bca6783894a431079f8a40515c328401875ae6195e4390d34b88" Nov 03 21:45:04.716326 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:04.716298 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm_odh-ai-gateway-infra(4f568d8c-8808-4c5c-b862-fcdc67e8aa7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" podUID="4f568d8c-8808-4c5c-b862-fcdc67e8aa7c" Nov 03 21:45:04.717982 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:04.717962 4963 scope.go:122] "RemoveContainer" containerID="ff875bd39efef49844db07dd43ccaf3e5a0c93bf3ccf1d517d949776b9425d9e" Nov 03 21:45:04.718248 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:04.718221 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5_odh-ai-gateway-infra(d98347d8-c132-4714-98fb-4642418cec20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" podUID="d98347d8-c132-4714-98fb-4642418cec20" Nov 03 21:45:04.719521 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:04.719500 4963 generic.go:356] "Generic (PLEG): container finished" podID="27132aa5-cb8f-45a4-9ca8-2998ba4f70dc" containerID="03ce6ef0cd3a2bb41d565705611827f4762ae96c2004914144b2cc52bf2e34d3" exitCode=60 Nov 03 21:45:04.719795 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:04.719575 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" event={"ID":"27132aa5-cb8f-45a4-9ca8-2998ba4f70dc","Type":"ContainerDied","Data":"03ce6ef0cd3a2bb41d565705611827f4762ae96c2004914144b2cc52bf2e34d3"} Nov 03 21:45:04.719862 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:04.719813 4963 scope.go:122] "RemoveContainer" containerID="03ce6ef0cd3a2bb41d565705611827f4762ae96c2004914144b2cc52bf2e34d3" Nov 03 21:45:04.720129 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:04.720105 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9_odh-ai-gateway-infra(27132aa5-cb8f-45a4-9ca8-2998ba4f70dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" podUID="27132aa5-cb8f-45a4-9ca8-2998ba4f70dc" Nov 03 21:45:04.721537 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:04.721513 4963 generic.go:356] "Generic (PLEG): container finished" podID="02bd2073-42ed-4acd-94b6-f3002d558c5a" containerID="c1955ab0ef3537793091405472ca85da9869c3e5b6b9f4d9134b8ef974428b5a" exitCode=60 Nov 03 21:45:04.721632 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:04.721581 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" event={"ID":"02bd2073-42ed-4acd-94b6-f3002d558c5a","Type":"ContainerDied","Data":"c1955ab0ef3537793091405472ca85da9869c3e5b6b9f4d9134b8ef974428b5a"} Nov 03 21:45:04.721858 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:04.721843 4963 scope.go:122] "RemoveContainer" containerID="c1955ab0ef3537793091405472ca85da9869c3e5b6b9f4d9134b8ef974428b5a" Nov 03 21:45:04.722132 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:04.722113 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89_odh-ai-gateway-infra(02bd2073-42ed-4acd-94b6-f3002d558c5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" podUID="02bd2073-42ed-4acd-94b6-f3002d558c5a" Nov 03 21:45:04.729887 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:04.729864 4963 scope.go:122] "RemoveContainer" containerID="383d686bb35345f352e4439061baf6788598a52c91169085e1926ff8edc6e419" Nov 03 21:45:04.743814 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:04.743787 4963 scope.go:122] "RemoveContainer" containerID="59ff283d4817167392cdf24401c23d2860f6574d4acc0cb00d54df75f2a1dd65" Nov 03 21:45:05.231568 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:05.231531 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:05.729543 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:05.729510 4963 scope.go:122] "RemoveContainer" containerID="5018661a9ab8bca6783894a431079f8a40515c328401875ae6195e4390d34b88" Nov 03 21:45:05.730055 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:05.729873 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm_odh-ai-gateway-infra(4f568d8c-8808-4c5c-b862-fcdc67e8aa7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" podUID="4f568d8c-8808-4c5c-b862-fcdc67e8aa7c" Nov 03 21:45:05.731219 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:05.731197 4963 scope.go:122] "RemoveContainer" containerID="03ce6ef0cd3a2bb41d565705611827f4762ae96c2004914144b2cc52bf2e34d3" Nov 03 21:45:05.731487 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:05.731466 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9_odh-ai-gateway-infra(27132aa5-cb8f-45a4-9ca8-2998ba4f70dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" podUID="27132aa5-cb8f-45a4-9ca8-2998ba4f70dc" Nov 03 21:45:05.732747 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:05.732731 4963 scope.go:122] "RemoveContainer" containerID="c1955ab0ef3537793091405472ca85da9869c3e5b6b9f4d9134b8ef974428b5a" Nov 03 21:45:05.732965 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:05.732947 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89_odh-ai-gateway-infra(02bd2073-42ed-4acd-94b6-f3002d558c5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" podUID="02bd2073-42ed-4acd-94b6-f3002d558c5a" Nov 03 21:45:08.292294 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:08.292258 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:09.046876 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:09.046830 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:09.248589 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:09.248559 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:09.830932 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:09.830874 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns"] Nov 03 21:45:09.836740 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:09.836711 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:45:09.891307 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:09.891265 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns"] Nov 03 21:45:09.922305 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:09.922257 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d504c953-1e7c-4605-a17b-fa5fa0dabed3-metrics-certs\") pod \"maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns\" (UID: \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:45:09.922467 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:09.922324 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65xzb\" (UniqueName: \"kubernetes.io/projected/d504c953-1e7c-4605-a17b-fa5fa0dabed3-kube-api-access-65xzb\") pod \"maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns\" (UID: \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:45:09.922567 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:09.922538 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d504c953-1e7c-4605-a17b-fa5fa0dabed3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns\" (UID: \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:45:09.922647 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:09.922600 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d504c953-1e7c-4605-a17b-fa5fa0dabed3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns\" (UID: \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:45:09.933761 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:09.933722 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-a02404-serving-cert\"" Nov 03 21:45:10.024289 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:10.024192 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d504c953-1e7c-4605-a17b-fa5fa0dabed3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns\" (UID: \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:45:10.024289 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:10.024243 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d504c953-1e7c-4605-a17b-fa5fa0dabed3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns\" (UID: \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:45:10.024506 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:10.024333 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d504c953-1e7c-4605-a17b-fa5fa0dabed3-metrics-certs\") pod \"maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns\" (UID: \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:45:10.024506 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:10.024366 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65xzb\" (UniqueName: \"kubernetes.io/projected/d504c953-1e7c-4605-a17b-fa5fa0dabed3-kube-api-access-65xzb\") pod \"maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns\" (UID: \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:45:10.024506 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:10.024439 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-serving-cert: secret "maas-api-e2e-extmod-b-a02404-serving-cert" not found Nov 03 21:45:10.024506 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:10.024505 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d504c953-1e7c-4605-a17b-fa5fa0dabed3-maas-api-tls podName:d504c953-1e7c-4605-a17b-fa5fa0dabed3 nodeName:}" failed. No retries permitted until 2026-11-03 21:45:10.524485743 +0000 UTC m=+3459.684557997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d504c953-1e7c-4605-a17b-fa5fa0dabed3-maas-api-tls") pod "maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" (UID: "d504c953-1e7c-4605-a17b-fa5fa0dabed3") : secret "maas-api-e2e-extmod-b-a02404-serving-cert" not found Nov 03 21:45:10.025071 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:10.025045 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d504c953-1e7c-4605-a17b-fa5fa0dabed3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns\" (UID: \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:45:10.027196 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:10.027162 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d504c953-1e7c-4605-a17b-fa5fa0dabed3-metrics-certs\") pod \"maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns\" (UID: \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:45:10.060160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:10.060130 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:10.083294 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:10.083227 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65xzb\" (UniqueName: \"kubernetes.io/projected/d504c953-1e7c-4605-a17b-fa5fa0dabed3-kube-api-access-65xzb\") pod \"maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns\" (UID: \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:45:10.250965 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:10.250932 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:10.440553 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:10.440466 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:10.529992 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:10.529946 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d504c953-1e7c-4605-a17b-fa5fa0dabed3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns\" (UID: \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:45:10.532692 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:10.532665 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d504c953-1e7c-4605-a17b-fa5fa0dabed3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns\" (UID: \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:45:10.751873 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:10.751831 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:45:11.130247 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:11.130163 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns"] Nov 03 21:45:11.137100 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:45:11.137072 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd504c953_1e7c_4605_a17b_fa5fa0dabed3.slice/crio-ef2a8451cee95d5800f60f78f58c609a9e020e2f33876290353dcbdf57348bd8 WatchSource:0}: Error finding container ef2a8451cee95d5800f60f78f58c609a9e020e2f33876290353dcbdf57348bd8: Status 404 returned error can't find the container with id ef2a8451cee95d5800f60f78f58c609a9e020e2f33876290353dcbdf57348bd8 Nov 03 21:45:11.774313 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:11.774271 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" event={"ID":"d504c953-1e7c-4605-a17b-fa5fa0dabed3","Type":"ContainerStarted","Data":"ef2a8451cee95d5800f60f78f58c609a9e020e2f33876290353dcbdf57348bd8"} Nov 03 21:45:12.529448 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:12.529404 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:45:12.782158 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:12.782062 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" event={"ID":"d504c953-1e7c-4605-a17b-fa5fa0dabed3","Type":"ContainerStarted","Data":"2f25ee7b37acf68f6e0fe4be6ee678ac0f42c587423383bd13f5afc8f0553a2b"} Nov 03 21:45:12.782158 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:12.782136 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:45:12.896867 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:12.896811 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" podStartSLOduration=2.971560284 podStartE2EDuration="3.896797224s" totalImagesPullingTime="925.23694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:12.893469369 +0000 UTC m=+3462.053541643" watchObservedRunningTime="2026-11-03 21:45:12.896797224 +0000 UTC m=+3462.056869538" Nov 03 21:45:13.345727 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:13.345692 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:13.537613 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:13.537579 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:14.960897 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:14.960861 4963 scope.go:122] "RemoveContainer" containerID="03ce6ef0cd3a2bb41d565705611827f4762ae96c2004914144b2cc52bf2e34d3" Nov 03 21:45:14.961388 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:14.960982 4963 scope.go:122] "RemoveContainer" containerID="c1955ab0ef3537793091405472ca85da9869c3e5b6b9f4d9134b8ef974428b5a" Nov 03 21:45:15.053417 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:15.053388 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:15.804430 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:15.804392 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" event={"ID":"27132aa5-cb8f-45a4-9ca8-2998ba4f70dc","Type":"ContainerStarted","Data":"4d38378fe6ea9b745c03f58b05eb16b8e27b8ecf57a42332185fdff379d187e0"} Nov 03 21:45:15.806306 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:15.806279 4963 generic.go:356] "Generic (PLEG): container finished" podID="02bd2073-42ed-4acd-94b6-f3002d558c5a" containerID="c009a8b561588256f10ba649c7665c7102032b51c3cf99439cf183c4a77c0bfe" exitCode=60 Nov 03 21:45:15.806403 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:15.806349 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" event={"ID":"02bd2073-42ed-4acd-94b6-f3002d558c5a","Type":"ContainerDied","Data":"c009a8b561588256f10ba649c7665c7102032b51c3cf99439cf183c4a77c0bfe"} Nov 03 21:45:15.806440 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:15.806403 4963 scope.go:122] "RemoveContainer" containerID="c1955ab0ef3537793091405472ca85da9869c3e5b6b9f4d9134b8ef974428b5a" Nov 03 21:45:15.806713 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:15.806696 4963 scope.go:122] "RemoveContainer" containerID="c009a8b561588256f10ba649c7665c7102032b51c3cf99439cf183c4a77c0bfe" Nov 03 21:45:15.806964 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:15.806943 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89_odh-ai-gateway-infra(02bd2073-42ed-4acd-94b6-f3002d558c5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" podUID="02bd2073-42ed-4acd-94b6-f3002d558c5a" Nov 03 21:45:15.819845 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:15.819819 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:15.960169 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:15.960135 4963 scope.go:122] "RemoveContainer" containerID="ff875bd39efef49844db07dd43ccaf3e5a0c93bf3ccf1d517d949776b9425d9e" Nov 03 21:45:16.814086 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:16.814048 4963 generic.go:356] "Generic (PLEG): container finished" podID="d98347d8-c132-4714-98fb-4642418cec20" containerID="70da81184a8174a54cfc95f987277d315aae3eb2fdf2aa7209181ca02c911301" exitCode=60 Nov 03 21:45:16.814486 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:16.814120 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" event={"ID":"d98347d8-c132-4714-98fb-4642418cec20","Type":"ContainerDied","Data":"70da81184a8174a54cfc95f987277d315aae3eb2fdf2aa7209181ca02c911301"} Nov 03 21:45:16.814486 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:16.814179 4963 scope.go:122] "RemoveContainer" containerID="ff875bd39efef49844db07dd43ccaf3e5a0c93bf3ccf1d517d949776b9425d9e" Nov 03 21:45:16.814592 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:16.814578 4963 scope.go:122] "RemoveContainer" containerID="70da81184a8174a54cfc95f987277d315aae3eb2fdf2aa7209181ca02c911301" Nov 03 21:45:16.815118 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:16.814884 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5_odh-ai-gateway-infra(d98347d8-c132-4714-98fb-4642418cec20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" podUID="d98347d8-c132-4714-98fb-4642418cec20" Nov 03 21:45:16.816278 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:16.816258 4963 generic.go:356] "Generic (PLEG): container finished" podID="27132aa5-cb8f-45a4-9ca8-2998ba4f70dc" containerID="4d38378fe6ea9b745c03f58b05eb16b8e27b8ecf57a42332185fdff379d187e0" exitCode=60 Nov 03 21:45:16.816356 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:16.816327 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" event={"ID":"27132aa5-cb8f-45a4-9ca8-2998ba4f70dc","Type":"ContainerDied","Data":"4d38378fe6ea9b745c03f58b05eb16b8e27b8ecf57a42332185fdff379d187e0"} Nov 03 21:45:16.816659 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:16.816645 4963 scope.go:122] "RemoveContainer" containerID="4d38378fe6ea9b745c03f58b05eb16b8e27b8ecf57a42332185fdff379d187e0" Nov 03 21:45:16.816977 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:16.816954 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9_odh-ai-gateway-infra(27132aa5-cb8f-45a4-9ca8-2998ba4f70dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" podUID="27132aa5-cb8f-45a4-9ca8-2998ba4f70dc" Nov 03 21:45:16.848225 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:16.848202 4963 scope.go:122] "RemoveContainer" containerID="03ce6ef0cd3a2bb41d565705611827f4762ae96c2004914144b2cc52bf2e34d3" Nov 03 21:45:16.930841 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:16.930805 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9"] Nov 03 21:45:16.961021 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:16.960987 4963 scope.go:122] "RemoveContainer" containerID="1f350c030f52050ae6bdc99dce68319754917f9ed7eb078fa494ddcdf800b68b" Nov 03 21:45:16.961166 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:16.961116 4963 scope.go:122] "RemoveContainer" containerID="5018661a9ab8bca6783894a431079f8a40515c328401875ae6195e4390d34b88" Nov 03 21:45:17.036676 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:17.036588 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89"] Nov 03 21:45:17.238837 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:17.238809 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" Nov 03 21:45:17.315125 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:17.315097 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02bd2073-42ed-4acd-94b6-f3002d558c5a-kube-api-access-24xnx\" (UniqueName: \"kubernetes.io/projected/02bd2073-42ed-4acd-94b6-f3002d558c5a-kube-api-access-24xnx\") pod \"02bd2073-42ed-4acd-94b6-f3002d558c5a\" (UID: \"02bd2073-42ed-4acd-94b6-f3002d558c5a\") " Nov 03 21:45:17.315272 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:17.315152 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02bd2073-42ed-4acd-94b6-f3002d558c5a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02bd2073-42ed-4acd-94b6-f3002d558c5a-maas-api-service-ca\") pod \"02bd2073-42ed-4acd-94b6-f3002d558c5a\" (UID: \"02bd2073-42ed-4acd-94b6-f3002d558c5a\") " Nov 03 21:45:17.315515 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:17.315492 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02bd2073-42ed-4acd-94b6-f3002d558c5a-maas-api-service-ca" pod "02bd2073-42ed-4acd-94b6-f3002d558c5a" (UID: "02bd2073-42ed-4acd-94b6-f3002d558c5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:17.315598 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:17.315580 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02bd2073-42ed-4acd-94b6-f3002d558c5a-maas-api-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:45:17.317408 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:17.317386 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bd2073-42ed-4acd-94b6-f3002d558c5a-kube-api-access-24xnx" pod "02bd2073-42ed-4acd-94b6-f3002d558c5a" (UID: "02bd2073-42ed-4acd-94b6-f3002d558c5a"). InnerVolumeSpecName "kube-api-access-24xnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:17.417214 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:17.417173 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24xnx\" (UniqueName: \"kubernetes.io/projected/02bd2073-42ed-4acd-94b6-f3002d558c5a-kube-api-access-24xnx\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:45:17.829088 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:17.829056 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" Nov 03 21:45:17.829088 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:17.829077 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89" event={"ID":"02bd2073-42ed-4acd-94b6-f3002d558c5a","Type":"ContainerDied","Data":"3f891d9ba4c4f5618b815f5d3df2f13c6f778609e39ea9021c03162a8c6d8032"} Nov 03 21:45:17.829522 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:17.829135 4963 scope.go:122] "RemoveContainer" containerID="c009a8b561588256f10ba649c7665c7102032b51c3cf99439cf183c4a77c0bfe" Nov 03 21:45:17.831179 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:17.831156 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" event={"ID":"c91983bf-8cf7-4259-99f2-1139c1c9a5c1","Type":"ContainerStarted","Data":"8137b7d64019453e47e3a3f29219fb32354750fab49058628fa2b804814ab2a4"} Nov 03 21:45:17.833173 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:17.833149 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" event={"ID":"4f568d8c-8808-4c5c-b862-fcdc67e8aa7c","Type":"ContainerStarted","Data":"23e763be6fd68504a00a50ae7e51b5be83e1aa428cb8b1dc54a0f34a02206644"} Nov 03 21:45:17.943628 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:17.943591 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5"] Nov 03 21:45:18.047193 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.047163 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" Nov 03 21:45:18.095852 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.095825 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" Nov 03 21:45:18.123590 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.123563 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27132aa5-cb8f-45a4-9ca8-2998ba4f70dc-kube-api-access-lrps5\" (UniqueName: \"kubernetes.io/projected/27132aa5-cb8f-45a4-9ca8-2998ba4f70dc-kube-api-access-lrps5\") pod \"27132aa5-cb8f-45a4-9ca8-2998ba4f70dc\" (UID: \"27132aa5-cb8f-45a4-9ca8-2998ba4f70dc\") " Nov 03 21:45:18.123796 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.123644 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27132aa5-cb8f-45a4-9ca8-2998ba4f70dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27132aa5-cb8f-45a4-9ca8-2998ba4f70dc-maas-api-service-ca\") pod \"27132aa5-cb8f-45a4-9ca8-2998ba4f70dc\" (UID: \"27132aa5-cb8f-45a4-9ca8-2998ba4f70dc\") " Nov 03 21:45:18.124044 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.124011 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27132aa5-cb8f-45a4-9ca8-2998ba4f70dc-maas-api-service-ca" pod "27132aa5-cb8f-45a4-9ca8-2998ba4f70dc" (UID: "27132aa5-cb8f-45a4-9ca8-2998ba4f70dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:18.126078 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.126056 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27132aa5-cb8f-45a4-9ca8-2998ba4f70dc-kube-api-access-lrps5" pod "27132aa5-cb8f-45a4-9ca8-2998ba4f70dc" (UID: "27132aa5-cb8f-45a4-9ca8-2998ba4f70dc"). InnerVolumeSpecName "kube-api-access-lrps5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:18.224783 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.224742 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d98347d8-c132-4714-98fb-4642418cec20-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98347d8-c132-4714-98fb-4642418cec20-maas-api-service-ca\") pod \"d98347d8-c132-4714-98fb-4642418cec20\" (UID: \"d98347d8-c132-4714-98fb-4642418cec20\") " Nov 03 21:45:18.225133 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.224892 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98347d8-c132-4714-98fb-4642418cec20-kube-api-access-xhk6h\" (UniqueName: \"kubernetes.io/projected/d98347d8-c132-4714-98fb-4642418cec20-kube-api-access-xhk6h\") pod \"d98347d8-c132-4714-98fb-4642418cec20\" (UID: \"d98347d8-c132-4714-98fb-4642418cec20\") " Nov 03 21:45:18.225251 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.225225 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98347d8-c132-4714-98fb-4642418cec20-maas-api-service-ca" pod "d98347d8-c132-4714-98fb-4642418cec20" (UID: "d98347d8-c132-4714-98fb-4642418cec20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:18.225456 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.225435 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98347d8-c132-4714-98fb-4642418cec20-maas-api-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:45:18.225516 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.225463 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrps5\" (UniqueName: \"kubernetes.io/projected/27132aa5-cb8f-45a4-9ca8-2998ba4f70dc-kube-api-access-lrps5\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:45:18.225516 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.225491 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27132aa5-cb8f-45a4-9ca8-2998ba4f70dc-maas-api-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:45:18.227519 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.227491 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98347d8-c132-4714-98fb-4642418cec20-kube-api-access-xhk6h" pod "d98347d8-c132-4714-98fb-4642418cec20" (UID: "d98347d8-c132-4714-98fb-4642418cec20"). InnerVolumeSpecName "kube-api-access-xhk6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:18.326863 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.326822 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhk6h\" (UniqueName: \"kubernetes.io/projected/d98347d8-c132-4714-98fb-4642418cec20-kube-api-access-xhk6h\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:45:18.342755 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.342635 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" podStartSLOduration=18.342617898 podStartE2EDuration="18.342617898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:18.340106212 +0000 UTC m=+3467.500178484" watchObservedRunningTime="2026-11-03 21:45:18.342617898 +0000 UTC m=+3467.502690171" Nov 03 21:45:18.350477 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.350446 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:18.452582 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.452283 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89"] Nov 03 21:45:18.457789 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.457754 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895705-92k89"] Nov 03 21:45:18.538478 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.538441 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:18.726709 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.726668 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd"] Nov 03 21:45:18.727476 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727451 4963 cpu_manager.go:414] "containerMap: removing container" podUID="d98347d8-c132-4714-98fb-4642418cec20" containerName="cleanup" Nov 03 21:45:18.727549 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727479 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98347d8-c132-4714-98fb-4642418cec20" containerName="cleanup" Nov 03 21:45:18.727549 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727516 4963 cpu_manager.go:414] "containerMap: removing container" podUID="27132aa5-cb8f-45a4-9ca8-2998ba4f70dc" containerName="cleanup" Nov 03 21:45:18.727549 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727524 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27132aa5-cb8f-45a4-9ca8-2998ba4f70dc" containerName="cleanup" Nov 03 21:45:18.727549 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727537 4963 cpu_manager.go:414] "containerMap: removing container" podUID="27132aa5-cb8f-45a4-9ca8-2998ba4f70dc" containerName="cleanup" Nov 03 21:45:18.727549 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727545 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27132aa5-cb8f-45a4-9ca8-2998ba4f70dc" containerName="cleanup" Nov 03 21:45:18.727705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727565 4963 cpu_manager.go:414] "containerMap: removing container" podUID="02bd2073-42ed-4acd-94b6-f3002d558c5a" containerName="cleanup" Nov 03 21:45:18.727705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727572 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02bd2073-42ed-4acd-94b6-f3002d558c5a" containerName="cleanup" Nov 03 21:45:18.727705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727584 4963 cpu_manager.go:414] "containerMap: removing container" podUID="27132aa5-cb8f-45a4-9ca8-2998ba4f70dc" containerName="cleanup" Nov 03 21:45:18.727705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727590 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27132aa5-cb8f-45a4-9ca8-2998ba4f70dc" containerName="cleanup" Nov 03 21:45:18.727705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727603 4963 cpu_manager.go:414] "containerMap: removing container" podUID="d98347d8-c132-4714-98fb-4642418cec20" containerName="cleanup" Nov 03 21:45:18.727705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727609 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98347d8-c132-4714-98fb-4642418cec20" containerName="cleanup" Nov 03 21:45:18.727705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727622 4963 cpu_manager.go:414] "containerMap: removing container" podUID="02bd2073-42ed-4acd-94b6-f3002d558c5a" containerName="cleanup" Nov 03 21:45:18.727705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727630 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02bd2073-42ed-4acd-94b6-f3002d558c5a" containerName="cleanup" Nov 03 21:45:18.727705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727663 4963 cpu_manager.go:414] "containerMap: removing container" podUID="02bd2073-42ed-4acd-94b6-f3002d558c5a" containerName="cleanup" Nov 03 21:45:18.727705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727670 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02bd2073-42ed-4acd-94b6-f3002d558c5a" containerName="cleanup" Nov 03 21:45:18.728007 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727821 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="02bd2073-42ed-4acd-94b6-f3002d558c5a" containerName="cleanup" Nov 03 21:45:18.728007 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727840 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98347d8-c132-4714-98fb-4642418cec20" containerName="cleanup" Nov 03 21:45:18.728007 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727854 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="27132aa5-cb8f-45a4-9ca8-2998ba4f70dc" containerName="cleanup" Nov 03 21:45:18.728007 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727864 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="27132aa5-cb8f-45a4-9ca8-2998ba4f70dc" containerName="cleanup" Nov 03 21:45:18.728007 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727874 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="27132aa5-cb8f-45a4-9ca8-2998ba4f70dc" containerName="cleanup" Nov 03 21:45:18.728007 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727887 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98347d8-c132-4714-98fb-4642418cec20" containerName="cleanup" Nov 03 21:45:18.728007 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.727894 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="02bd2073-42ed-4acd-94b6-f3002d558c5a" containerName="cleanup" Nov 03 21:45:18.731577 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.731555 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd" Nov 03 21:45:18.782224 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.782185 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-6a186d-dockercfg-wlj9l\"" Nov 03 21:45:18.782405 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.782286 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6a186d\"" Nov 03 21:45:18.786090 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.786060 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd"] Nov 03 21:45:18.795230 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.795203 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:45:18.832704 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.832656 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krppq\" (UniqueName: \"kubernetes.io/projected/c2e484f1-6067-45f8-a14e-e43fcfeab191-kube-api-access-krppq\") pod \"payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd\" (UID: \"c2e484f1-6067-45f8-a14e-e43fcfeab191\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd" Nov 03 21:45:18.833281 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.832990 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e484f1-6067-45f8-a14e-e43fcfeab191-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd\" (UID: \"c2e484f1-6067-45f8-a14e-e43fcfeab191\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd" Nov 03 21:45:18.844967 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.844937 4963 generic.go:356] "Generic (PLEG): container finished" podID="c91983bf-8cf7-4259-99f2-1139c1c9a5c1" containerID="8137b7d64019453e47e3a3f29219fb32354750fab49058628fa2b804814ab2a4" exitCode=60 Nov 03 21:45:18.845134 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.845055 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" event={"ID":"c91983bf-8cf7-4259-99f2-1139c1c9a5c1","Type":"ContainerDied","Data":"8137b7d64019453e47e3a3f29219fb32354750fab49058628fa2b804814ab2a4"} Nov 03 21:45:18.845134 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.845096 4963 scope.go:122] "RemoveContainer" containerID="1f350c030f52050ae6bdc99dce68319754917f9ed7eb078fa494ddcdf800b68b" Nov 03 21:45:18.845600 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.845504 4963 scope.go:122] "RemoveContainer" containerID="8137b7d64019453e47e3a3f29219fb32354750fab49058628fa2b804814ab2a4" Nov 03 21:45:18.845814 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:18.845790 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p_odh-ai-gateway-infra(c91983bf-8cf7-4259-99f2-1139c1c9a5c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" podUID="c91983bf-8cf7-4259-99f2-1139c1c9a5c1" Nov 03 21:45:18.847984 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.847961 4963 generic.go:356] "Generic (PLEG): container finished" podID="4f568d8c-8808-4c5c-b862-fcdc67e8aa7c" containerID="23e763be6fd68504a00a50ae7e51b5be83e1aa428cb8b1dc54a0f34a02206644" exitCode=60 Nov 03 21:45:18.848094 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.848054 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" event={"ID":"4f568d8c-8808-4c5c-b862-fcdc67e8aa7c","Type":"ContainerDied","Data":"23e763be6fd68504a00a50ae7e51b5be83e1aa428cb8b1dc54a0f34a02206644"} Nov 03 21:45:18.848688 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.848667 4963 scope.go:122] "RemoveContainer" containerID="23e763be6fd68504a00a50ae7e51b5be83e1aa428cb8b1dc54a0f34a02206644" Nov 03 21:45:18.849425 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:18.849044 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm_odh-ai-gateway-infra(4f568d8c-8808-4c5c-b862-fcdc67e8aa7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" podUID="4f568d8c-8808-4c5c-b862-fcdc67e8aa7c" Nov 03 21:45:18.850000 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.849964 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" Nov 03 21:45:18.850340 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.849961 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5" event={"ID":"d98347d8-c132-4714-98fb-4642418cec20","Type":"ContainerDied","Data":"831c0d19d3f486281849d44be88d6b3a9795e50790ab3342a1d0882f01089b13"} Nov 03 21:45:18.852016 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.851611 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" event={"ID":"27132aa5-cb8f-45a4-9ca8-2998ba4f70dc","Type":"ContainerDied","Data":"143f1bba6f0999869fa2bc642019e737a904c0d14fcc6d743e847dd20d663b0d"} Nov 03 21:45:18.852016 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.851708 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9" Nov 03 21:45:18.865192 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.865164 4963 scope.go:122] "RemoveContainer" containerID="5018661a9ab8bca6783894a431079f8a40515c328401875ae6195e4390d34b88" Nov 03 21:45:18.881933 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.881896 4963 scope.go:122] "RemoveContainer" containerID="70da81184a8174a54cfc95f987277d315aae3eb2fdf2aa7209181ca02c911301" Nov 03 21:45:18.892994 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.892975 4963 scope.go:122] "RemoveContainer" containerID="4d38378fe6ea9b745c03f58b05eb16b8e27b8ecf57a42332185fdff379d187e0" Nov 03 21:45:18.934785 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.934742 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e484f1-6067-45f8-a14e-e43fcfeab191-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd\" (UID: \"c2e484f1-6067-45f8-a14e-e43fcfeab191\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd" Nov 03 21:45:18.935310 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.935276 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krppq\" (UniqueName: \"kubernetes.io/projected/c2e484f1-6067-45f8-a14e-e43fcfeab191-kube-api-access-krppq\") pod \"payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd\" (UID: \"c2e484f1-6067-45f8-a14e-e43fcfeab191\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd" Nov 03 21:45:18.935832 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.935801 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e484f1-6067-45f8-a14e-e43fcfeab191-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd\" (UID: \"c2e484f1-6067-45f8-a14e-e43fcfeab191\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd" Nov 03 21:45:18.970170 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.970130 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02bd2073-42ed-4acd-94b6-f3002d558c5a" path="/var/lib/kubelet/pods/02bd2073-42ed-4acd-94b6-f3002d558c5a/volumes" Nov 03 21:45:18.994671 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:18.994576 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krppq\" (UniqueName: \"kubernetes.io/projected/c2e484f1-6067-45f8-a14e-e43fcfeab191-kube-api-access-krppq\") pod \"payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd\" (UID: \"c2e484f1-6067-45f8-a14e-e43fcfeab191\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd" Nov 03 21:45:19.054486 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.054450 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd" Nov 03 21:45:19.062633 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.062605 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5"] Nov 03 21:45:19.066767 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.066736 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895705-bpgz5"] Nov 03 21:45:19.232912 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.232863 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm"] Nov 03 21:45:19.353868 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.353779 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p"] Nov 03 21:45:19.389534 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.389496 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9"] Nov 03 21:45:19.395159 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.395129 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a3dc77-29895705-8j4w9"] Nov 03 21:45:19.466775 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.466743 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd"] Nov 03 21:45:19.472160 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:45:19.472128 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e484f1_6067_45f8_a14e_e43fcfeab191.slice/crio-ac29a6af667960ea079a09b84c28a80e5258c8aabad8f8eac4f31f1219be89b9 WatchSource:0}: Error finding container ac29a6af667960ea079a09b84c28a80e5258c8aabad8f8eac4f31f1219be89b9: Status 404 returned error can't find the container with id ac29a6af667960ea079a09b84c28a80e5258c8aabad8f8eac4f31f1219be89b9 Nov 03 21:45:19.863110 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.863071 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd" event={"ID":"c2e484f1-6067-45f8-a14e-e43fcfeab191","Type":"ContainerStarted","Data":"05b41ed187aba783f762d787ec666309da3582647548330470cd12a72e3390bc"} Nov 03 21:45:19.863593 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.863135 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd" event={"ID":"c2e484f1-6067-45f8-a14e-e43fcfeab191","Type":"ContainerStarted","Data":"ac29a6af667960ea079a09b84c28a80e5258c8aabad8f8eac4f31f1219be89b9"} Nov 03 21:45:19.863593 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.863161 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd" Nov 03 21:45:19.894209 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.894177 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7"] Nov 03 21:45:19.894779 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.894764 4963 cpu_manager.go:414] "containerMap: removing container" podUID="d98347d8-c132-4714-98fb-4642418cec20" containerName="cleanup" Nov 03 21:45:19.894828 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.894781 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98347d8-c132-4714-98fb-4642418cec20" containerName="cleanup" Nov 03 21:45:19.894949 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.894936 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98347d8-c132-4714-98fb-4642418cec20" containerName="cleanup" Nov 03 21:45:19.894988 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.894954 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="02bd2073-42ed-4acd-94b6-f3002d558c5a" containerName="cleanup" Nov 03 21:45:19.900116 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.900090 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:45:19.965218 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.959150 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-7da0f7-serving-cert\"" Nov 03 21:45:19.974209 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.974140 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7"] Nov 03 21:45:19.981727 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:19.981656 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd" podStartSLOduration=1.9816322560000001 podStartE2EDuration="1.981632256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:19.975821126 +0000 UTC m=+3469.135893404" watchObservedRunningTime="2026-11-03 21:45:19.981632256 +0000 UTC m=+3469.141704530" Nov 03 21:45:20.047280 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.047238 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a297c4b6-bcfb-4225-9651-5d69ec8b6715-metrics-certs\") pod \"maas-api-e2e-autores-7da0f7-775745f69c-kctz7\" (UID: \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:45:20.047565 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.047543 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a297c4b6-bcfb-4225-9651-5d69ec8b6715-maas-api-tls\") pod \"maas-api-e2e-autores-7da0f7-775745f69c-kctz7\" (UID: \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:45:20.047797 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.047777 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a297c4b6-bcfb-4225-9651-5d69ec8b6715-openshift-service-ca\") pod \"maas-api-e2e-autores-7da0f7-775745f69c-kctz7\" (UID: \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:45:20.047991 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.047971 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl75k\" (UniqueName: \"kubernetes.io/projected/a297c4b6-bcfb-4225-9651-5d69ec8b6715-kube-api-access-vl75k\") pod \"maas-api-e2e-autores-7da0f7-775745f69c-kctz7\" (UID: \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:45:20.076572 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.076544 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" Nov 03 21:45:20.080356 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.080330 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" Nov 03 21:45:20.121694 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.121606 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:20.149015 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.148981 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c91983bf-8cf7-4259-99f2-1139c1c9a5c1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91983bf-8cf7-4259-99f2-1139c1c9a5c1-maas-api-service-ca\") pod \"c91983bf-8cf7-4259-99f2-1139c1c9a5c1\" (UID: \"c91983bf-8cf7-4259-99f2-1139c1c9a5c1\") " Nov 03 21:45:20.149219 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.149044 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c91983bf-8cf7-4259-99f2-1139c1c9a5c1-kube-api-access-qrnmd\" (UniqueName: \"kubernetes.io/projected/c91983bf-8cf7-4259-99f2-1139c1c9a5c1-kube-api-access-qrnmd\") pod \"c91983bf-8cf7-4259-99f2-1139c1c9a5c1\" (UID: \"c91983bf-8cf7-4259-99f2-1139c1c9a5c1\") " Nov 03 21:45:20.149219 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.149165 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f568d8c-8808-4c5c-b862-fcdc67e8aa7c-kube-api-access-9x56k\" (UniqueName: \"kubernetes.io/projected/4f568d8c-8808-4c5c-b862-fcdc67e8aa7c-kube-api-access-9x56k\") pod \"4f568d8c-8808-4c5c-b862-fcdc67e8aa7c\" (UID: \"4f568d8c-8808-4c5c-b862-fcdc67e8aa7c\") " Nov 03 21:45:20.149333 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.149306 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f568d8c-8808-4c5c-b862-fcdc67e8aa7c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f568d8c-8808-4c5c-b862-fcdc67e8aa7c-maas-api-service-ca\") pod \"4f568d8c-8808-4c5c-b862-fcdc67e8aa7c\" (UID: \"4f568d8c-8808-4c5c-b862-fcdc67e8aa7c\") " Nov 03 21:45:20.149487 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.149437 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c91983bf-8cf7-4259-99f2-1139c1c9a5c1-maas-api-service-ca" pod "c91983bf-8cf7-4259-99f2-1139c1c9a5c1" (UID: "c91983bf-8cf7-4259-99f2-1139c1c9a5c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:20.149702 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.149678 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f568d8c-8808-4c5c-b862-fcdc67e8aa7c-maas-api-service-ca" pod "4f568d8c-8808-4c5c-b862-fcdc67e8aa7c" (UID: "4f568d8c-8808-4c5c-b862-fcdc67e8aa7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:20.149770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.149701 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a297c4b6-bcfb-4225-9651-5d69ec8b6715-metrics-certs\") pod \"maas-api-e2e-autores-7da0f7-775745f69c-kctz7\" (UID: \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:45:20.149770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.149741 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a297c4b6-bcfb-4225-9651-5d69ec8b6715-maas-api-tls\") pod \"maas-api-e2e-autores-7da0f7-775745f69c-kctz7\" (UID: \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:45:20.149884 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.149864 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a297c4b6-bcfb-4225-9651-5d69ec8b6715-openshift-service-ca\") pod \"maas-api-e2e-autores-7da0f7-775745f69c-kctz7\" (UID: \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:45:20.149969 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.149895 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl75k\" (UniqueName: \"kubernetes.io/projected/a297c4b6-bcfb-4225-9651-5d69ec8b6715-kube-api-access-vl75k\") pod \"maas-api-e2e-autores-7da0f7-775745f69c-kctz7\" (UID: \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:45:20.149969 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:20.149932 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-serving-cert: secret "maas-api-e2e-autores-7da0f7-serving-cert" not found Nov 03 21:45:20.150073 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:20.150001 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a297c4b6-bcfb-4225-9651-5d69ec8b6715-maas-api-tls podName:a297c4b6-bcfb-4225-9651-5d69ec8b6715 nodeName:}" failed. No retries permitted until 2026-11-03 21:45:20.649977741 +0000 UTC m=+3469.810049992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a297c4b6-bcfb-4225-9651-5d69ec8b6715-maas-api-tls") pod "maas-api-e2e-autores-7da0f7-775745f69c-kctz7" (UID: "a297c4b6-bcfb-4225-9651-5d69ec8b6715") : secret "maas-api-e2e-autores-7da0f7-serving-cert" not found Nov 03 21:45:20.150217 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.150188 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f568d8c-8808-4c5c-b862-fcdc67e8aa7c-maas-api-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:45:20.150326 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.150221 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91983bf-8cf7-4259-99f2-1139c1c9a5c1-maas-api-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:45:20.150643 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.150620 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a297c4b6-bcfb-4225-9651-5d69ec8b6715-openshift-service-ca\") pod \"maas-api-e2e-autores-7da0f7-775745f69c-kctz7\" (UID: \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:45:20.151623 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.151605 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91983bf-8cf7-4259-99f2-1139c1c9a5c1-kube-api-access-qrnmd" pod "c91983bf-8cf7-4259-99f2-1139c1c9a5c1" (UID: "c91983bf-8cf7-4259-99f2-1139c1c9a5c1"). InnerVolumeSpecName "kube-api-access-qrnmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:20.151713 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.151632 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f568d8c-8808-4c5c-b862-fcdc67e8aa7c-kube-api-access-9x56k" pod "4f568d8c-8808-4c5c-b862-fcdc67e8aa7c" (UID: "4f568d8c-8808-4c5c-b862-fcdc67e8aa7c"). InnerVolumeSpecName "kube-api-access-9x56k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:20.152211 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.152188 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a297c4b6-bcfb-4225-9651-5d69ec8b6715-metrics-certs\") pod \"maas-api-e2e-autores-7da0f7-775745f69c-kctz7\" (UID: \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:45:20.209253 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.209215 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl75k\" (UniqueName: \"kubernetes.io/projected/a297c4b6-bcfb-4225-9651-5d69ec8b6715-kube-api-access-vl75k\") pod \"maas-api-e2e-autores-7da0f7-775745f69c-kctz7\" (UID: \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:45:20.251739 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.251705 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x56k\" (UniqueName: \"kubernetes.io/projected/4f568d8c-8808-4c5c-b862-fcdc67e8aa7c-kube-api-access-9x56k\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:45:20.251739 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.251738 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrnmd\" (UniqueName: \"kubernetes.io/projected/c91983bf-8cf7-4259-99f2-1139c1c9a5c1-kube-api-access-qrnmd\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:45:20.311067 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.311028 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:20.500823 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.500773 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:20.657966 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.657898 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a297c4b6-bcfb-4225-9651-5d69ec8b6715-maas-api-tls\") pod \"maas-api-e2e-autores-7da0f7-775745f69c-kctz7\" (UID: \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:45:20.660370 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.660341 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a297c4b6-bcfb-4225-9651-5d69ec8b6715-maas-api-tls\") pod \"maas-api-e2e-autores-7da0f7-775745f69c-kctz7\" (UID: \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:45:20.814851 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.814765 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:45:20.877888 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.877859 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" Nov 03 21:45:20.878658 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.877953 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p" event={"ID":"c91983bf-8cf7-4259-99f2-1139c1c9a5c1","Type":"ContainerDied","Data":"4a644ca9a82a4634d8dda4d25995613037697ab7f87e2fc1f57df63e10958ec5"} Nov 03 21:45:20.878658 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.878038 4963 scope.go:122] "RemoveContainer" containerID="8137b7d64019453e47e3a3f29219fb32354750fab49058628fa2b804814ab2a4" Nov 03 21:45:20.889194 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.882446 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" Nov 03 21:45:20.889194 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.882511 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm" event={"ID":"4f568d8c-8808-4c5c-b862-fcdc67e8aa7c","Type":"ContainerDied","Data":"d3c32f9246f803fcc356aaa7cadf21476b6bf4efc6cfa1ef0c038a07ae2ef8ad"} Nov 03 21:45:20.896375 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.896335 4963 scope.go:122] "RemoveContainer" containerID="23e763be6fd68504a00a50ae7e51b5be83e1aa428cb8b1dc54a0f34a02206644" Nov 03 21:45:20.969131 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.969095 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27132aa5-cb8f-45a4-9ca8-2998ba4f70dc" path="/var/lib/kubelet/pods/27132aa5-cb8f-45a4-9ca8-2998ba4f70dc/volumes" Nov 03 21:45:20.969644 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:20.969621 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d98347d8-c132-4714-98fb-4642418cec20" path="/var/lib/kubelet/pods/d98347d8-c132-4714-98fb-4642418cec20/volumes" Nov 03 21:45:21.039317 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:21.039113 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm"] Nov 03 21:45:21.046039 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:21.046006 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e2db-29895705-c22wm"] Nov 03 21:45:21.147750 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:21.147655 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p"] Nov 03 21:45:21.155466 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:21.155430 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895705-7997p"] Nov 03 21:45:21.194651 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:21.194615 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7"] Nov 03 21:45:21.200012 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:45:21.199980 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda297c4b6_bcfb_4225_9651_5d69ec8b6715.slice/crio-60a01234d497bab414843778dd8c31ddef3837453023999150617cad0a39cae9 WatchSource:0}: Error finding container 60a01234d497bab414843778dd8c31ddef3837453023999150617cad0a39cae9: Status 404 returned error can't find the container with id 60a01234d497bab414843778dd8c31ddef3837453023999150617cad0a39cae9 Nov 03 21:45:21.894234 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:21.894191 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" event={"ID":"a297c4b6-bcfb-4225-9651-5d69ec8b6715","Type":"ContainerStarted","Data":"60a01234d497bab414843778dd8c31ddef3837453023999150617cad0a39cae9"} Nov 03 21:45:22.906777 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:22.906738 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" event={"ID":"a297c4b6-bcfb-4225-9651-5d69ec8b6715","Type":"ContainerStarted","Data":"d5c87a75226f5754a611c3522a1bdd16eb82e60f8e402cfd104dfdfa819ea642"} Nov 03 21:45:22.907246 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:22.906941 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:45:22.969103 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:22.969069 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f568d8c-8808-4c5c-b862-fcdc67e8aa7c" path="/var/lib/kubelet/pods/4f568d8c-8808-4c5c-b862-fcdc67e8aa7c/volumes" Nov 03 21:45:22.969546 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:22.969532 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c91983bf-8cf7-4259-99f2-1139c1c9a5c1" path="/var/lib/kubelet/pods/c91983bf-8cf7-4259-99f2-1139c1c9a5c1/volumes" Nov 03 21:45:23.019371 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:23.019166 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" podStartSLOduration=3.127181915 podStartE2EDuration="4.019145508s" totalImagesPullingTime="891.963593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:23.017802824 +0000 UTC m=+3472.177875097" watchObservedRunningTime="2026-11-03 21:45:23.019145508 +0000 UTC m=+3472.179217782" Nov 03 21:45:25.138634 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:25.138591 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:28.918408 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:28.918370 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:45:30.071463 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:30.071427 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:30.886731 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:30.886695 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd" Nov 03 21:45:32.704884 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:32.704852 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:34.755460 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:34.755424 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:38.730390 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:38.730351 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:39.109364 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:39.109261 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks"] Nov 03 21:45:39.219675 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:39.219629 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk"] Nov 03 21:45:39.523209 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:39.523167 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:39.707106 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:39.707065 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:39.899219 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:39.899141 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:40.649540 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:40.649504 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:42.528513 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:42.528475 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:45:44.110585 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:44.110516 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks" podUID="ebf2301a-0ad8-449f-a2ff-7888665c112b" containerName="payload-processing" containerID="cri-o://aab967a31ac532247ed2dbccd0e703b2d785a2c235911d2829359abc43e90296" gracePeriod=25 Nov 03 21:45:44.220932 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:44.220846 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk" podUID="aa6ce46d-6643-423e-b237-e26dc6eb00ca" containerName="payload-pre-processing" containerID="cri-o://c9e989e6cfe9169bd03c83866d29071a13f6941532f86627fb83471120f49caa" gracePeriod=25 Nov 03 21:45:44.395636 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:44.395607 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks" Nov 03 21:45:44.469264 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:44.469223 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:44.525114 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:44.525084 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk" Nov 03 21:45:44.539710 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:44.539674 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebf2301a-0ad8-449f-a2ff-7888665c112b-kube-api-access-lrcvs\" (UniqueName: \"kubernetes.io/projected/ebf2301a-0ad8-449f-a2ff-7888665c112b-kube-api-access-lrcvs\") pod \"ebf2301a-0ad8-449f-a2ff-7888665c112b\" (UID: \"ebf2301a-0ad8-449f-a2ff-7888665c112b\") " Nov 03 21:45:44.539955 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:44.539885 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebf2301a-0ad8-449f-a2ff-7888665c112b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf2301a-0ad8-449f-a2ff-7888665c112b-plugins-config-volume\") pod \"ebf2301a-0ad8-449f-a2ff-7888665c112b\" (UID: \"ebf2301a-0ad8-449f-a2ff-7888665c112b\") " Nov 03 21:45:44.540967 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:44.540938 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf2301a-0ad8-449f-a2ff-7888665c112b-plugins-config-volume" pod "ebf2301a-0ad8-449f-a2ff-7888665c112b" (UID: "ebf2301a-0ad8-449f-a2ff-7888665c112b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:44.542593 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:44.542571 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf2301a-0ad8-449f-a2ff-7888665c112b-kube-api-access-lrcvs" pod "ebf2301a-0ad8-449f-a2ff-7888665c112b" (UID: "ebf2301a-0ad8-449f-a2ff-7888665c112b"). InnerVolumeSpecName "kube-api-access-lrcvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:44.544991 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:44.544077 4963 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf2301a-0ad8-449f-a2ff-7888665c112b-plugins-config-volume\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:45:44.544991 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:44.544104 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrcvs\" (UniqueName: \"kubernetes.io/projected/ebf2301a-0ad8-449f-a2ff-7888665c112b-kube-api-access-lrcvs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:45:44.646972 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:44.645365 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa6ce46d-6643-423e-b237-e26dc6eb00ca-kube-api-access-bj6wm\" (UniqueName: \"kubernetes.io/projected/aa6ce46d-6643-423e-b237-e26dc6eb00ca-kube-api-access-bj6wm\") pod \"aa6ce46d-6643-423e-b237-e26dc6eb00ca\" (UID: \"aa6ce46d-6643-423e-b237-e26dc6eb00ca\") " Nov 03 21:45:44.646972 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:44.645491 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa6ce46d-6643-423e-b237-e26dc6eb00ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6ce46d-6643-423e-b237-e26dc6eb00ca-plugins-config-volume\") pod \"aa6ce46d-6643-423e-b237-e26dc6eb00ca\" (UID: \"aa6ce46d-6643-423e-b237-e26dc6eb00ca\") " Nov 03 21:45:44.646972 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:44.645867 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6ce46d-6643-423e-b237-e26dc6eb00ca-plugins-config-volume" pod "aa6ce46d-6643-423e-b237-e26dc6eb00ca" (UID: "aa6ce46d-6643-423e-b237-e26dc6eb00ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:44.646972 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:44.646074 4963 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6ce46d-6643-423e-b237-e26dc6eb00ca-plugins-config-volume\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:45:44.647952 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:44.647892 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6ce46d-6643-423e-b237-e26dc6eb00ca-kube-api-access-bj6wm" pod "aa6ce46d-6643-423e-b237-e26dc6eb00ca" (UID: "aa6ce46d-6643-423e-b237-e26dc6eb00ca"). InnerVolumeSpecName "kube-api-access-bj6wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:44.652223 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:44.652199 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:44.747083 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:44.747046 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj6wm\" (UniqueName: \"kubernetes.io/projected/aa6ce46d-6643-423e-b237-e26dc6eb00ca-kube-api-access-bj6wm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:45:45.064230 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.064198 4963 generic.go:356] "Generic (PLEG): container finished" podID="ebf2301a-0ad8-449f-a2ff-7888665c112b" containerID="aab967a31ac532247ed2dbccd0e703b2d785a2c235911d2829359abc43e90296" exitCode=0 Nov 03 21:45:45.064426 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.064243 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks" Nov 03 21:45:45.064426 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.064288 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks" event={"ID":"ebf2301a-0ad8-449f-a2ff-7888665c112b","Type":"ContainerDied","Data":"aab967a31ac532247ed2dbccd0e703b2d785a2c235911d2829359abc43e90296"} Nov 03 21:45:45.064426 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.064339 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks" event={"ID":"ebf2301a-0ad8-449f-a2ff-7888665c112b","Type":"ContainerDied","Data":"09aaa0af8923632a07a2e15b8566916e348825d85973521c5d63d76c2d25d373"} Nov 03 21:45:45.064426 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.064364 4963 scope.go:122] "RemoveContainer" containerID="aab967a31ac532247ed2dbccd0e703b2d785a2c235911d2829359abc43e90296" Nov 03 21:45:45.065748 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.065719 4963 generic.go:356] "Generic (PLEG): container finished" podID="aa6ce46d-6643-423e-b237-e26dc6eb00ca" containerID="c9e989e6cfe9169bd03c83866d29071a13f6941532f86627fb83471120f49caa" exitCode=0 Nov 03 21:45:45.065835 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.065766 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk" Nov 03 21:45:45.065835 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.065802 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk" event={"ID":"aa6ce46d-6643-423e-b237-e26dc6eb00ca","Type":"ContainerDied","Data":"c9e989e6cfe9169bd03c83866d29071a13f6941532f86627fb83471120f49caa"} Nov 03 21:45:45.065990 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.065848 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk" event={"ID":"aa6ce46d-6643-423e-b237-e26dc6eb00ca","Type":"ContainerDied","Data":"d708e271cfd8709b0377da6972f407b52136018c7126f54594d6927a2080c354"} Nov 03 21:45:45.074860 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.074836 4963 scope.go:122] "RemoveContainer" containerID="aab967a31ac532247ed2dbccd0e703b2d785a2c235911d2829359abc43e90296" Nov 03 21:45:45.075218 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:45.075192 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aab967a31ac532247ed2dbccd0e703b2d785a2c235911d2829359abc43e90296\": container with ID starting with aab967a31ac532247ed2dbccd0e703b2d785a2c235911d2829359abc43e90296 not found: ID does not exist" containerID="aab967a31ac532247ed2dbccd0e703b2d785a2c235911d2829359abc43e90296" Nov 03 21:45:45.075312 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.075225 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aab967a31ac532247ed2dbccd0e703b2d785a2c235911d2829359abc43e90296"} err="failed to get container status \"aab967a31ac532247ed2dbccd0e703b2d785a2c235911d2829359abc43e90296\": rpc error: code = NotFound desc = could not find container \"aab967a31ac532247ed2dbccd0e703b2d785a2c235911d2829359abc43e90296\": container with ID starting with aab967a31ac532247ed2dbccd0e703b2d785a2c235911d2829359abc43e90296 not found: ID does not exist" Nov 03 21:45:45.075312 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.075245 4963 scope.go:122] "RemoveContainer" containerID="c9e989e6cfe9169bd03c83866d29071a13f6941532f86627fb83471120f49caa" Nov 03 21:45:45.088015 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.087993 4963 scope.go:122] "RemoveContainer" containerID="c9e989e6cfe9169bd03c83866d29071a13f6941532f86627fb83471120f49caa" Nov 03 21:45:45.088362 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:45:45.088339 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e989e6cfe9169bd03c83866d29071a13f6941532f86627fb83471120f49caa\": container with ID starting with c9e989e6cfe9169bd03c83866d29071a13f6941532f86627fb83471120f49caa not found: ID does not exist" containerID="c9e989e6cfe9169bd03c83866d29071a13f6941532f86627fb83471120f49caa" Nov 03 21:45:45.088421 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.088371 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e989e6cfe9169bd03c83866d29071a13f6941532f86627fb83471120f49caa"} err="failed to get container status \"c9e989e6cfe9169bd03c83866d29071a13f6941532f86627fb83471120f49caa\": rpc error: code = NotFound desc = could not find container \"c9e989e6cfe9169bd03c83866d29071a13f6941532f86627fb83471120f49caa\": container with ID starting with c9e989e6cfe9169bd03c83866d29071a13f6941532f86627fb83471120f49caa not found: ID does not exist" Nov 03 21:45:45.220720 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.220677 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks"] Nov 03 21:45:45.268256 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.268162 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-130a8c-5df7dfc6f6-7nkks"] Nov 03 21:45:45.330277 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.330238 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk"] Nov 03 21:45:45.334678 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.334652 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-130a8c-5769fb95ff-xxsnk"] Nov 03 21:45:45.722143 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:45.722113 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:46.969430 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:46.969389 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa6ce46d-6643-423e-b237-e26dc6eb00ca" path="/var/lib/kubelet/pods/aa6ce46d-6643-423e-b237-e26dc6eb00ca/volumes" Nov 03 21:45:46.970077 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:46.970056 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebf2301a-0ad8-449f-a2ff-7888665c112b" path="/var/lib/kubelet/pods/ebf2301a-0ad8-449f-a2ff-7888665c112b/volumes" Nov 03 21:45:47.166559 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:47.166527 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:47.366658 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:47.366571 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:49.427462 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:49.427429 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:51.521946 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:51.521900 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:54.009164 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:54.009123 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:55.067344 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:55.067310 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:45:58.634477 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:45:58.634440 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:03.012604 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:03.012571 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:06.825338 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:06.825298 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:07.011203 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:07.011161 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:07.469588 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:07.469553 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:11.954785 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:11.954750 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:12.528043 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:46:12.528004 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:46:13.068411 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.068379 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk"] Nov 03 21:46:13.068975 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.068958 4963 cpu_manager.go:414] "containerMap: removing container" podUID="c91983bf-8cf7-4259-99f2-1139c1c9a5c1" containerName="cleanup" Nov 03 21:46:13.068975 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.068976 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c91983bf-8cf7-4259-99f2-1139c1c9a5c1" containerName="cleanup" Nov 03 21:46:13.069090 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.068995 4963 cpu_manager.go:414] "containerMap: removing container" podUID="aa6ce46d-6643-423e-b237-e26dc6eb00ca" containerName="payload-pre-processing" Nov 03 21:46:13.069090 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.069004 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa6ce46d-6643-423e-b237-e26dc6eb00ca" containerName="payload-pre-processing" Nov 03 21:46:13.069090 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.069018 4963 cpu_manager.go:414] "containerMap: removing container" podUID="4f568d8c-8808-4c5c-b862-fcdc67e8aa7c" containerName="cleanup" Nov 03 21:46:13.069090 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.069025 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f568d8c-8808-4c5c-b862-fcdc67e8aa7c" containerName="cleanup" Nov 03 21:46:13.069090 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.069046 4963 cpu_manager.go:414] "containerMap: removing container" podUID="ebf2301a-0ad8-449f-a2ff-7888665c112b" containerName="payload-processing" Nov 03 21:46:13.069090 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.069051 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf2301a-0ad8-449f-a2ff-7888665c112b" containerName="payload-processing" Nov 03 21:46:13.069090 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.069060 4963 cpu_manager.go:414] "containerMap: removing container" podUID="c91983bf-8cf7-4259-99f2-1139c1c9a5c1" containerName="cleanup" Nov 03 21:46:13.069090 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.069065 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c91983bf-8cf7-4259-99f2-1139c1c9a5c1" containerName="cleanup" Nov 03 21:46:13.069090 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.069089 4963 cpu_manager.go:414] "containerMap: removing container" podUID="4f568d8c-8808-4c5c-b862-fcdc67e8aa7c" containerName="cleanup" Nov 03 21:46:13.069090 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.069094 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f568d8c-8808-4c5c-b862-fcdc67e8aa7c" containerName="cleanup" Nov 03 21:46:13.069527 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.069101 4963 cpu_manager.go:414] "containerMap: removing container" podUID="4f568d8c-8808-4c5c-b862-fcdc67e8aa7c" containerName="cleanup" Nov 03 21:46:13.069527 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.069105 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f568d8c-8808-4c5c-b862-fcdc67e8aa7c" containerName="cleanup" Nov 03 21:46:13.069527 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.069224 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="c91983bf-8cf7-4259-99f2-1139c1c9a5c1" containerName="cleanup" Nov 03 21:46:13.069527 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.069234 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa6ce46d-6643-423e-b237-e26dc6eb00ca" containerName="payload-pre-processing" Nov 03 21:46:13.069527 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.069240 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f568d8c-8808-4c5c-b862-fcdc67e8aa7c" containerName="cleanup" Nov 03 21:46:13.069527 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.069246 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf2301a-0ad8-449f-a2ff-7888665c112b" containerName="payload-processing" Nov 03 21:46:13.069527 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.069252 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="c91983bf-8cf7-4259-99f2-1139c1c9a5c1" containerName="cleanup" Nov 03 21:46:13.069527 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.069261 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f568d8c-8808-4c5c-b862-fcdc67e8aa7c" containerName="cleanup" Nov 03 21:46:13.069527 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.069267 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f568d8c-8808-4c5c-b862-fcdc67e8aa7c" containerName="cleanup" Nov 03 21:46:13.072717 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.072700 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.122698 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.122662 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-7da0f7-openshift-default-dockercfg-h89nd\"" Nov 03 21:46:13.165396 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.165363 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-data\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.165613 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.165403 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-workload-certs\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.165613 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.165551 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-credential-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.165735 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.165624 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9a143a-3fb6-4824-8d4e-cca981204821-istiod-ca-cert\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.165735 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.165665 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdfsb\" (UniqueName: \"kubernetes.io/projected/8a9a143a-3fb6-4824-8d4e-cca981204821-kube-api-access-wdfsb\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.165735 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.165700 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-envoy\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.165868 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.165754 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-workload-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.165868 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.165791 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-podinfo\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.165994 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.165873 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-token\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.266816 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.266780 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-workload-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.267056 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.266835 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-podinfo\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.267056 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.266861 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-token\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.267056 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.267024 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-data\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.267250 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.267058 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-workload-certs\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.267250 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.267197 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-credential-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.267361 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.267259 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9a143a-3fb6-4824-8d4e-cca981204821-istiod-ca-cert\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.267361 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.267285 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdfsb\" (UniqueName: \"kubernetes.io/projected/8a9a143a-3fb6-4824-8d4e-cca981204821-kube-api-access-wdfsb\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.267361 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.267305 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-workload-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.267361 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.267325 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-envoy\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.267565 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.267370 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-workload-certs\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.267565 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.267438 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-data\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.267565 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.267521 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-credential-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.268130 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.268112 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9a143a-3fb6-4824-8d4e-cca981204821-istiod-ca-cert\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.269713 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.269678 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-envoy\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.269831 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.269809 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-podinfo\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.273641 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.273614 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk"] Nov 03 21:46:13.325483 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.325396 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-token\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.325720 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.325694 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdfsb\" (UniqueName: \"kubernetes.io/projected/8a9a143a-3fb6-4824-8d4e-cca981204821-kube-api-access-wdfsb\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.386399 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.386354 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:13.819982 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.819944 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk"] Nov 03 21:46:13.825078 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:46:13.825046 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9a143a_3fb6_4824_8d4e_cca981204821.slice/crio-6583d3a844230abae8bc08220fca1cda7d3b855d2f578c608cda8b73c684716b WatchSource:0}: Error finding container 6583d3a844230abae8bc08220fca1cda7d3b855d2f578c608cda8b73c684716b: Status 404 returned error can't find the container with id 6583d3a844230abae8bc08220fca1cda7d3b855d2f578c608cda8b73c684716b Nov 03 21:46:13.827769 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.827736 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:13.827847 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.827811 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:13.827887 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:13.827856 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:14.258280 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:14.258241 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" event={"ID":"8a9a143a-3fb6-4824-8d4e-cca981204821","Type":"ContainerStarted","Data":"b2b0cf79e734136647651cd0fac8b70f1514cdbaa16191e61ee9a94db4a9f57e"} Nov 03 21:46:14.258685 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:14.258301 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" event={"ID":"8a9a143a-3fb6-4824-8d4e-cca981204821","Type":"ContainerStarted","Data":"6583d3a844230abae8bc08220fca1cda7d3b855d2f578c608cda8b73c684716b"} Nov 03 21:46:14.392930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:14.390070 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:14.392930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:14.392655 4963 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:46:14.392930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:14.392734 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" podUID="8a9a143a-3fb6-4824-8d4e-cca981204821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 21:46:14.422793 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:14.422732 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" podStartSLOduration=1.422717051 podStartE2EDuration="1.422717051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:14.420126376 +0000 UTC m=+3523.580198652" watchObservedRunningTime="2026-11-03 21:46:14.422717051 +0000 UTC m=+3523.582789399" Nov 03 21:46:15.402930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:15.397319 4963 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:46:15.402930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:15.397384 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" podUID="8a9a143a-3fb6-4824-8d4e-cca981204821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 21:46:16.395930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:16.392350 4963 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:46:16.395930 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:16.392413 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" podUID="8a9a143a-3fb6-4824-8d4e-cca981204821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 21:46:16.606821 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:16.606773 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:17.395083 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:17.395040 4963 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:46:17.395413 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:17.395383 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" podUID="8a9a143a-3fb6-4824-8d4e-cca981204821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 21:46:18.395406 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:18.395363 4963 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:46:18.396054 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:18.396013 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" podUID="8a9a143a-3fb6-4824-8d4e-cca981204821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 21:46:19.392359 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:19.392317 4963 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:46:19.392696 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:19.392668 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" podUID="8a9a143a-3fb6-4824-8d4e-cca981204821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 21:46:20.392222 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:20.392188 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:20.392693 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:20.392673 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:20.393365 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:20.393333 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:46:21.128720 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:21.128687 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:25.596354 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:25.596322 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:30.113422 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:30.113386 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:34.602096 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:34.602063 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:39.071783 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:39.071748 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:40.980531 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:40.980451 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:42.528132 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:46:42.528087 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:46:42.904373 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:42.904281 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7"] Nov 03 21:46:42.904575 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:42.904524 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" podUID="ea16191a-c781-4ea4-a759-909277f85cc3" containerName="maas-api" containerID="cri-o://573fc2737244542b109417cbfdee6ca74e84a4fa2b6f86d71f4330da79a9a4f0" gracePeriod=30 Nov 03 21:46:43.045574 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.045540 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:43.194359 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.194326 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:46:43.255232 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.255191 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea16191a-c781-4ea4-a759-909277f85cc3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea16191a-c781-4ea4-a759-909277f85cc3-openshift-service-ca\") pod \"ea16191a-c781-4ea4-a759-909277f85cc3\" (UID: \"ea16191a-c781-4ea4-a759-909277f85cc3\") " Nov 03 21:46:43.255446 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.255273 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea16191a-c781-4ea4-a759-909277f85cc3-kube-api-access-g4t7g\" (UniqueName: \"kubernetes.io/projected/ea16191a-c781-4ea4-a759-909277f85cc3-kube-api-access-g4t7g\") pod \"ea16191a-c781-4ea4-a759-909277f85cc3\" (UID: \"ea16191a-c781-4ea4-a759-909277f85cc3\") " Nov 03 21:46:43.255446 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.255296 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea16191a-c781-4ea4-a759-909277f85cc3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea16191a-c781-4ea4-a759-909277f85cc3-maas-api-tls\") pod \"ea16191a-c781-4ea4-a759-909277f85cc3\" (UID: \"ea16191a-c781-4ea4-a759-909277f85cc3\") " Nov 03 21:46:43.255446 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.255322 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea16191a-c781-4ea4-a759-909277f85cc3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea16191a-c781-4ea4-a759-909277f85cc3-metrics-certs\") pod \"ea16191a-c781-4ea4-a759-909277f85cc3\" (UID: \"ea16191a-c781-4ea4-a759-909277f85cc3\") " Nov 03 21:46:43.255631 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.255592 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea16191a-c781-4ea4-a759-909277f85cc3-openshift-service-ca" pod "ea16191a-c781-4ea4-a759-909277f85cc3" (UID: "ea16191a-c781-4ea4-a759-909277f85cc3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:46:43.255932 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.255897 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea16191a-c781-4ea4-a759-909277f85cc3-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:46:43.257557 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.257523 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea16191a-c781-4ea4-a759-909277f85cc3-maas-api-tls" pod "ea16191a-c781-4ea4-a759-909277f85cc3" (UID: "ea16191a-c781-4ea4-a759-909277f85cc3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:46:43.258018 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.257991 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea16191a-c781-4ea4-a759-909277f85cc3-metrics-certs" pod "ea16191a-c781-4ea4-a759-909277f85cc3" (UID: "ea16191a-c781-4ea4-a759-909277f85cc3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:46:43.258140 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.258002 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea16191a-c781-4ea4-a759-909277f85cc3-kube-api-access-g4t7g" pod "ea16191a-c781-4ea4-a759-909277f85cc3" (UID: "ea16191a-c781-4ea4-a759-909277f85cc3"). InnerVolumeSpecName "kube-api-access-g4t7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:46:43.309083 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.309035 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:43.357019 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.356982 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4t7g\" (UniqueName: \"kubernetes.io/projected/ea16191a-c781-4ea4-a759-909277f85cc3-kube-api-access-g4t7g\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:46:43.357019 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.357013 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea16191a-c781-4ea4-a759-909277f85cc3-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:46:43.357019 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.357025 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea16191a-c781-4ea4-a759-909277f85cc3-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:46:43.471065 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.471025 4963 generic.go:356] "Generic (PLEG): container finished" podID="ea16191a-c781-4ea4-a759-909277f85cc3" containerID="573fc2737244542b109417cbfdee6ca74e84a4fa2b6f86d71f4330da79a9a4f0" exitCode=0 Nov 03 21:46:43.471282 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.471079 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" Nov 03 21:46:43.471282 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.471079 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" event={"ID":"ea16191a-c781-4ea4-a759-909277f85cc3","Type":"ContainerDied","Data":"573fc2737244542b109417cbfdee6ca74e84a4fa2b6f86d71f4330da79a9a4f0"} Nov 03 21:46:43.471282 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.471146 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" event={"ID":"ea16191a-c781-4ea4-a759-909277f85cc3","Type":"ContainerDied","Data":"c72c60cf543920dd3407612c0d8c4dc91bd9579df0b16fd67c2d9f3de1291607"} Nov 03 21:46:43.471282 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.471181 4963 scope.go:122] "RemoveContainer" containerID="573fc2737244542b109417cbfdee6ca74e84a4fa2b6f86d71f4330da79a9a4f0" Nov 03 21:46:43.484878 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.484846 4963 scope.go:122] "RemoveContainer" containerID="573fc2737244542b109417cbfdee6ca74e84a4fa2b6f86d71f4330da79a9a4f0" Nov 03 21:46:43.485332 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:46:43.485312 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573fc2737244542b109417cbfdee6ca74e84a4fa2b6f86d71f4330da79a9a4f0\": container with ID starting with 573fc2737244542b109417cbfdee6ca74e84a4fa2b6f86d71f4330da79a9a4f0 not found: ID does not exist" containerID="573fc2737244542b109417cbfdee6ca74e84a4fa2b6f86d71f4330da79a9a4f0" Nov 03 21:46:43.485405 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.485341 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573fc2737244542b109417cbfdee6ca74e84a4fa2b6f86d71f4330da79a9a4f0"} err="failed to get container status \"573fc2737244542b109417cbfdee6ca74e84a4fa2b6f86d71f4330da79a9a4f0\": rpc error: code = NotFound desc = could not find container \"573fc2737244542b109417cbfdee6ca74e84a4fa2b6f86d71f4330da79a9a4f0\": container with ID starting with 573fc2737244542b109417cbfdee6ca74e84a4fa2b6f86d71f4330da79a9a4f0 not found: ID does not exist" Nov 03 21:46:43.628351 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.628316 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7"] Nov 03 21:46:43.632087 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:43.632063 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7"] Nov 03 21:46:44.397333 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:44.397296 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:44.585383 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:44.585321 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:44.782738 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:44.782706 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:44.967751 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:44.967715 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea16191a-c781-4ea4-a759-909277f85cc3" path="/var/lib/kubelet/pods/ea16191a-c781-4ea4-a759-909277f85cc3/volumes" Nov 03 21:46:44.983501 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:44.983467 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:45.947166 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:45.947080 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:46.123059 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:46.122996 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-130a8c-cb66b6968-5clf7" podUID="ea16191a-c781-4ea4-a759-909277f85cc3" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.123:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:46.545374 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:46.545340 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:47.123971 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:47.123893 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:47.340257 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:47.340211 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:49.335255 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.335218 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh"] Nov 03 21:46:49.335895 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.335870 4963 cpu_manager.go:414] "containerMap: removing container" podUID="ea16191a-c781-4ea4-a759-909277f85cc3" containerName="maas-api" Nov 03 21:46:49.335895 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.335888 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea16191a-c781-4ea4-a759-909277f85cc3" containerName="maas-api" Nov 03 21:46:49.336026 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.335914 4963 cpu_manager.go:414] "containerMap: removing container" podUID="c91983bf-8cf7-4259-99f2-1139c1c9a5c1" containerName="cleanup" Nov 03 21:46:49.336026 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.335922 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c91983bf-8cf7-4259-99f2-1139c1c9a5c1" containerName="cleanup" Nov 03 21:46:49.336100 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.336056 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea16191a-c781-4ea4-a759-909277f85cc3" containerName="maas-api" Nov 03 21:46:49.336100 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.336069 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="c91983bf-8cf7-4259-99f2-1139c1c9a5c1" containerName="cleanup" Nov 03 21:46:49.339639 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.339618 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:46:49.390199 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.390152 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-6a186d-serving-cert\"" Nov 03 21:46:49.402756 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.402720 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:49.428742 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.428708 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2acbef8-8a3b-45f1-ac55-15212a819222-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh\" (UID: \"f2acbef8-8a3b-45f1-ac55-15212a819222\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:46:49.428938 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.428778 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2acbef8-8a3b-45f1-ac55-15212a819222-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh\" (UID: \"f2acbef8-8a3b-45f1-ac55-15212a819222\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:46:49.429024 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.428966 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2acbef8-8a3b-45f1-ac55-15212a819222-metrics-certs\") pod \"maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh\" (UID: \"f2acbef8-8a3b-45f1-ac55-15212a819222\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:46:49.429024 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.429015 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjqtf\" (UniqueName: \"kubernetes.io/projected/f2acbef8-8a3b-45f1-ac55-15212a819222-kube-api-access-fjqtf\") pod \"maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh\" (UID: \"f2acbef8-8a3b-45f1-ac55-15212a819222\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:46:49.440505 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.440474 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh"] Nov 03 21:46:49.530845 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.530802 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2acbef8-8a3b-45f1-ac55-15212a819222-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh\" (UID: \"f2acbef8-8a3b-45f1-ac55-15212a819222\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:46:49.531080 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.530860 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2acbef8-8a3b-45f1-ac55-15212a819222-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh\" (UID: \"f2acbef8-8a3b-45f1-ac55-15212a819222\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:46:49.531080 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.530922 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2acbef8-8a3b-45f1-ac55-15212a819222-metrics-certs\") pod \"maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh\" (UID: \"f2acbef8-8a3b-45f1-ac55-15212a819222\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:46:49.531080 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.530947 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqtf\" (UniqueName: \"kubernetes.io/projected/f2acbef8-8a3b-45f1-ac55-15212a819222-kube-api-access-fjqtf\") pod \"maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh\" (UID: \"f2acbef8-8a3b-45f1-ac55-15212a819222\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:46:49.531080 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:46:49.531019 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-serving-cert: secret "maas-api-e2e-extmod-b-6a186d-serving-cert" not found Nov 03 21:46:49.531349 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:46:49.531103 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2acbef8-8a3b-45f1-ac55-15212a819222-maas-api-tls podName:f2acbef8-8a3b-45f1-ac55-15212a819222 nodeName:}" failed. No retries permitted until 2026-11-03 21:46:50.03108548 +0000 UTC m=+3559.191157731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2acbef8-8a3b-45f1-ac55-15212a819222-maas-api-tls") pod "maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" (UID: "f2acbef8-8a3b-45f1-ac55-15212a819222") : secret "maas-api-e2e-extmod-b-6a186d-serving-cert" not found Nov 03 21:46:49.531506 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.531485 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2acbef8-8a3b-45f1-ac55-15212a819222-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh\" (UID: \"f2acbef8-8a3b-45f1-ac55-15212a819222\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:46:49.533496 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.533464 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2acbef8-8a3b-45f1-ac55-15212a819222-metrics-certs\") pod \"maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh\" (UID: \"f2acbef8-8a3b-45f1-ac55-15212a819222\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:46:49.586650 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.586559 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:49.589628 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.589604 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjqtf\" (UniqueName: \"kubernetes.io/projected/f2acbef8-8a3b-45f1-ac55-15212a819222-kube-api-access-fjqtf\") pod \"maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh\" (UID: \"f2acbef8-8a3b-45f1-ac55-15212a819222\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:46:49.779819 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:49.779784 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:50.038149 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:50.038110 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2acbef8-8a3b-45f1-ac55-15212a819222-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh\" (UID: \"f2acbef8-8a3b-45f1-ac55-15212a819222\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:46:50.040647 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:50.040614 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2acbef8-8a3b-45f1-ac55-15212a819222-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh\" (UID: \"f2acbef8-8a3b-45f1-ac55-15212a819222\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:46:50.253581 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:50.253552 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:46:50.627031 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:50.626950 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh"] Nov 03 21:46:50.632077 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:46:50.632047 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2acbef8_8a3b_45f1_ac55_15212a819222.slice/crio-bf767f3e8ab8aa83be70b31f92af19af9011db590c64f3febe4e098d1062a113 WatchSource:0}: Error finding container bf767f3e8ab8aa83be70b31f92af19af9011db590c64f3febe4e098d1062a113: Status 404 returned error can't find the container with id bf767f3e8ab8aa83be70b31f92af19af9011db590c64f3febe4e098d1062a113 Nov 03 21:46:50.633838 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:50.633820 4963 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:46:51.171097 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:51.171059 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:51.355547 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:51.355518 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:51.525647 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:51.525607 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" event={"ID":"f2acbef8-8a3b-45f1-ac55-15212a819222","Type":"ContainerStarted","Data":"bf767f3e8ab8aa83be70b31f92af19af9011db590c64f3febe4e098d1062a113"} Nov 03 21:46:52.536986 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:52.536944 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" event={"ID":"f2acbef8-8a3b-45f1-ac55-15212a819222","Type":"ContainerStarted","Data":"e3073fef26aa157acb8ac5ac8e4bbc31a695f53aff1420a17c2b12ae3bacb9b7"} Nov 03 21:46:52.537461 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:52.537133 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:46:52.649976 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:52.649885 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" podStartSLOduration=2.660173469 podStartE2EDuration="3.649873034s" totalImagesPullingTime="989.699565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:52.649139773 +0000 UTC m=+3561.809212046" watchObservedRunningTime="2026-11-03 21:46:52.649873034 +0000 UTC m=+3561.809945306" Nov 03 21:46:53.777259 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:53.777224 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:54.116220 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:54.116144 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:54.504966 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:54.504923 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:55.117081 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:55.117051 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:56.879887 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:56.879852 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:58.549463 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:58.549431 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:46:58.693173 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:58.693135 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:58.725262 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:58.725221 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7"] Nov 03 21:46:58.725564 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:58.725507 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" podUID="a297c4b6-bcfb-4225-9651-5d69ec8b6715" containerName="maas-api" containerID="cri-o://d5c87a75226f5754a611c3522a1bdd16eb82e60f8e402cfd104dfdfa819ea642" gracePeriod=30 Nov 03 21:46:58.873836 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:58.873790 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:46:59.025428 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.025395 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:46:59.066106 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.066068 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a297c4b6-bcfb-4225-9651-5d69ec8b6715-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a297c4b6-bcfb-4225-9651-5d69ec8b6715-openshift-service-ca\") pod \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\" (UID: \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\") " Nov 03 21:46:59.066316 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.066122 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a297c4b6-bcfb-4225-9651-5d69ec8b6715-kube-api-access-vl75k\" (UniqueName: \"kubernetes.io/projected/a297c4b6-bcfb-4225-9651-5d69ec8b6715-kube-api-access-vl75k\") pod \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\" (UID: \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\") " Nov 03 21:46:59.066316 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.066211 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a297c4b6-bcfb-4225-9651-5d69ec8b6715-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a297c4b6-bcfb-4225-9651-5d69ec8b6715-metrics-certs\") pod \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\" (UID: \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\") " Nov 03 21:46:59.066316 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.066263 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a297c4b6-bcfb-4225-9651-5d69ec8b6715-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a297c4b6-bcfb-4225-9651-5d69ec8b6715-maas-api-tls\") pod \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\" (UID: \"a297c4b6-bcfb-4225-9651-5d69ec8b6715\") " Nov 03 21:46:59.066519 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.066493 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a297c4b6-bcfb-4225-9651-5d69ec8b6715-openshift-service-ca" pod "a297c4b6-bcfb-4225-9651-5d69ec8b6715" (UID: "a297c4b6-bcfb-4225-9651-5d69ec8b6715"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:46:59.066941 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.066880 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a297c4b6-bcfb-4225-9651-5d69ec8b6715-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:46:59.068556 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.068532 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a297c4b6-bcfb-4225-9651-5d69ec8b6715-maas-api-tls" pod "a297c4b6-bcfb-4225-9651-5d69ec8b6715" (UID: "a297c4b6-bcfb-4225-9651-5d69ec8b6715"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:46:59.068970 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.068944 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a297c4b6-bcfb-4225-9651-5d69ec8b6715-metrics-certs" pod "a297c4b6-bcfb-4225-9651-5d69ec8b6715" (UID: "a297c4b6-bcfb-4225-9651-5d69ec8b6715"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:46:59.069065 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.069018 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a297c4b6-bcfb-4225-9651-5d69ec8b6715-kube-api-access-vl75k" pod "a297c4b6-bcfb-4225-9651-5d69ec8b6715" (UID: "a297c4b6-bcfb-4225-9651-5d69ec8b6715"). InnerVolumeSpecName "kube-api-access-vl75k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:46:59.168257 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.168107 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl75k\" (UniqueName: \"kubernetes.io/projected/a297c4b6-bcfb-4225-9651-5d69ec8b6715-kube-api-access-vl75k\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:46:59.168257 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.168150 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a297c4b6-bcfb-4225-9651-5d69ec8b6715-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:46:59.169845 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.168166 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a297c4b6-bcfb-4225-9651-5d69ec8b6715-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:46:59.582941 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.582866 4963 generic.go:356] "Generic (PLEG): container finished" podID="a297c4b6-bcfb-4225-9651-5d69ec8b6715" containerID="d5c87a75226f5754a611c3522a1bdd16eb82e60f8e402cfd104dfdfa819ea642" exitCode=0 Nov 03 21:46:59.583384 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.582946 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" Nov 03 21:46:59.583384 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.582954 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" event={"ID":"a297c4b6-bcfb-4225-9651-5d69ec8b6715","Type":"ContainerDied","Data":"d5c87a75226f5754a611c3522a1bdd16eb82e60f8e402cfd104dfdfa819ea642"} Nov 03 21:46:59.583384 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.583012 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" event={"ID":"a297c4b6-bcfb-4225-9651-5d69ec8b6715","Type":"ContainerDied","Data":"60a01234d497bab414843778dd8c31ddef3837453023999150617cad0a39cae9"} Nov 03 21:46:59.583384 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.583033 4963 scope.go:122] "RemoveContainer" containerID="d5c87a75226f5754a611c3522a1bdd16eb82e60f8e402cfd104dfdfa819ea642" Nov 03 21:46:59.594264 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.594243 4963 scope.go:122] "RemoveContainer" containerID="d5c87a75226f5754a611c3522a1bdd16eb82e60f8e402cfd104dfdfa819ea642" Nov 03 21:46:59.594574 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:46:59.594551 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c87a75226f5754a611c3522a1bdd16eb82e60f8e402cfd104dfdfa819ea642\": container with ID starting with d5c87a75226f5754a611c3522a1bdd16eb82e60f8e402cfd104dfdfa819ea642 not found: ID does not exist" containerID="d5c87a75226f5754a611c3522a1bdd16eb82e60f8e402cfd104dfdfa819ea642" Nov 03 21:46:59.594630 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.594585 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c87a75226f5754a611c3522a1bdd16eb82e60f8e402cfd104dfdfa819ea642"} err="failed to get container status \"d5c87a75226f5754a611c3522a1bdd16eb82e60f8e402cfd104dfdfa819ea642\": rpc error: code = NotFound desc = could not find container \"d5c87a75226f5754a611c3522a1bdd16eb82e60f8e402cfd104dfdfa819ea642\": container with ID starting with d5c87a75226f5754a611c3522a1bdd16eb82e60f8e402cfd104dfdfa819ea642 not found: ID does not exist" Nov 03 21:46:59.740565 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.740522 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7"] Nov 03 21:46:59.787662 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.787626 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7"] Nov 03 21:46:59.870332 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:46:59.870253 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:00.597949 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:00.597894 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:00.975527 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:00.975488 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a297c4b6-bcfb-4225-9651-5d69ec8b6715" path="/var/lib/kubelet/pods/a297c4b6-bcfb-4225-9651-5d69ec8b6715/volumes" Nov 03 21:47:01.913834 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:01.913775 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7da0f7-775745f69c-kctz7" podUID="a297c4b6-bcfb-4225-9651-5d69ec8b6715" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.136:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:04.684468 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:04.684431 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss"] Nov 03 21:47:04.685017 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:04.684718 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" podUID="16279a5e-a28a-44a6-ba3c-05566f417084" containerName="maas-api" containerID="cri-o://6d66892a6c407c099ff7037ce7932705283f4b4e399da260aa96ec879b707723" gracePeriod=30 Nov 03 21:47:04.828820 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:04.828782 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:04.976811 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:04.976778 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:47:05.036741 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.036695 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16279a5e-a28a-44a6-ba3c-05566f417084-kube-api-access-s2g8c\" (UniqueName: \"kubernetes.io/projected/16279a5e-a28a-44a6-ba3c-05566f417084-kube-api-access-s2g8c\") pod \"16279a5e-a28a-44a6-ba3c-05566f417084\" (UID: \"16279a5e-a28a-44a6-ba3c-05566f417084\") " Nov 03 21:47:05.036961 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.036746 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16279a5e-a28a-44a6-ba3c-05566f417084-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16279a5e-a28a-44a6-ba3c-05566f417084-metrics-certs\") pod \"16279a5e-a28a-44a6-ba3c-05566f417084\" (UID: \"16279a5e-a28a-44a6-ba3c-05566f417084\") " Nov 03 21:47:05.036961 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.036857 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16279a5e-a28a-44a6-ba3c-05566f417084-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16279a5e-a28a-44a6-ba3c-05566f417084-maas-api-tls\") pod \"16279a5e-a28a-44a6-ba3c-05566f417084\" (UID: \"16279a5e-a28a-44a6-ba3c-05566f417084\") " Nov 03 21:47:05.037055 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.037032 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16279a5e-a28a-44a6-ba3c-05566f417084-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16279a5e-a28a-44a6-ba3c-05566f417084-openshift-service-ca\") pod \"16279a5e-a28a-44a6-ba3c-05566f417084\" (UID: \"16279a5e-a28a-44a6-ba3c-05566f417084\") " Nov 03 21:47:05.037405 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.037377 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16279a5e-a28a-44a6-ba3c-05566f417084-openshift-service-ca" pod "16279a5e-a28a-44a6-ba3c-05566f417084" (UID: "16279a5e-a28a-44a6-ba3c-05566f417084"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:05.037552 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.037539 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16279a5e-a28a-44a6-ba3c-05566f417084-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:05.039294 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.039263 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16279a5e-a28a-44a6-ba3c-05566f417084-kube-api-access-s2g8c" pod "16279a5e-a28a-44a6-ba3c-05566f417084" (UID: "16279a5e-a28a-44a6-ba3c-05566f417084"). InnerVolumeSpecName "kube-api-access-s2g8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:05.039448 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.039422 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16279a5e-a28a-44a6-ba3c-05566f417084-metrics-certs" pod "16279a5e-a28a-44a6-ba3c-05566f417084" (UID: "16279a5e-a28a-44a6-ba3c-05566f417084"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:05.039519 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.039451 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16279a5e-a28a-44a6-ba3c-05566f417084-maas-api-tls" pod "16279a5e-a28a-44a6-ba3c-05566f417084" (UID: "16279a5e-a28a-44a6-ba3c-05566f417084"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:05.139017 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.138980 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2g8c\" (UniqueName: \"kubernetes.io/projected/16279a5e-a28a-44a6-ba3c-05566f417084-kube-api-access-s2g8c\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:05.139017 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.139014 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16279a5e-a28a-44a6-ba3c-05566f417084-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:05.139017 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.139024 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16279a5e-a28a-44a6-ba3c-05566f417084-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:05.623589 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.623556 4963 generic.go:356] "Generic (PLEG): container finished" podID="16279a5e-a28a-44a6-ba3c-05566f417084" containerID="6d66892a6c407c099ff7037ce7932705283f4b4e399da260aa96ec879b707723" exitCode=0 Nov 03 21:47:05.623783 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.623604 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" Nov 03 21:47:05.623783 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.623611 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" event={"ID":"16279a5e-a28a-44a6-ba3c-05566f417084","Type":"ContainerDied","Data":"6d66892a6c407c099ff7037ce7932705283f4b4e399da260aa96ec879b707723"} Nov 03 21:47:05.623783 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.623655 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss" event={"ID":"16279a5e-a28a-44a6-ba3c-05566f417084","Type":"ContainerDied","Data":"22949dadc08d5286763d5e210799d49ca418a5eba050997178f0a4419f976dea"} Nov 03 21:47:05.623783 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.623685 4963 scope.go:122] "RemoveContainer" containerID="6d66892a6c407c099ff7037ce7932705283f4b4e399da260aa96ec879b707723" Nov 03 21:47:05.669131 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.653090 4963 scope.go:122] "RemoveContainer" containerID="6d66892a6c407c099ff7037ce7932705283f4b4e399da260aa96ec879b707723" Nov 03 21:47:05.669771 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:47:05.669617 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d66892a6c407c099ff7037ce7932705283f4b4e399da260aa96ec879b707723\": container with ID starting with 6d66892a6c407c099ff7037ce7932705283f4b4e399da260aa96ec879b707723 not found: ID does not exist" containerID="6d66892a6c407c099ff7037ce7932705283f4b4e399da260aa96ec879b707723" Nov 03 21:47:05.669771 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.669665 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d66892a6c407c099ff7037ce7932705283f4b4e399da260aa96ec879b707723"} err="failed to get container status \"6d66892a6c407c099ff7037ce7932705283f4b4e399da260aa96ec879b707723\": rpc error: code = NotFound desc = could not find container \"6d66892a6c407c099ff7037ce7932705283f4b4e399da260aa96ec879b707723\": container with ID starting with 6d66892a6c407c099ff7037ce7932705283f4b4e399da260aa96ec879b707723 not found: ID does not exist" Nov 03 21:47:05.784213 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.784170 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss"] Nov 03 21:47:05.788675 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.788624 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e2db-68b8988b8c-zczss"] Nov 03 21:47:05.829500 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:05.829456 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:06.023002 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:06.022967 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:06.652789 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:06.652752 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:06.967605 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:06.967559 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16279a5e-a28a-44a6-ba3c-05566f417084" path="/var/lib/kubelet/pods/16279a5e-a28a-44a6-ba3c-05566f417084/volumes" Nov 03 21:47:07.352900 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:07.352823 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:07.856359 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:07.856314 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk"] Nov 03 21:47:07.856599 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:07.856573 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" podUID="8a9a143a-3fb6-4824-8d4e-cca981204821" containerName="istio-proxy" containerID="cri-o://b2b0cf79e734136647651cd0fac8b70f1514cdbaa16191e61ee9a94db4a9f57e" gracePeriod=30 Nov 03 21:47:07.893622 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:07.893589 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:08.610829 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:08.610777 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:09.866024 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:09.865988 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:10.051837 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:10.051803 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:10.636941 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:10.636895 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:10.820664 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:10.820628 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:11.502783 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:11.502743 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:11.740941 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:11.740899 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:12.247929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:12.247872 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:12.529017 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:47:12.528930 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:47:13.142137 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.142063 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:47:13.260611 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.260578 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-workload-socket\") pod \"8a9a143a-3fb6-4824-8d4e-cca981204821\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " Nov 03 21:47:13.260784 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.260625 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-envoy\") pod \"8a9a143a-3fb6-4824-8d4e-cca981204821\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " Nov 03 21:47:13.260784 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.260672 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-podinfo\") pod \"8a9a143a-3fb6-4824-8d4e-cca981204821\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " Nov 03 21:47:13.260784 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.260746 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-token\" (UniqueName: \"kubernetes.io/projected/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-token\") pod \"8a9a143a-3fb6-4824-8d4e-cca981204821\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " Nov 03 21:47:13.261042 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.260789 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-data\") pod \"8a9a143a-3fb6-4824-8d4e-cca981204821\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " Nov 03 21:47:13.261042 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.260900 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-workload-socket" pod "8a9a143a-3fb6-4824-8d4e-cca981204821" (UID: "8a9a143a-3fb6-4824-8d4e-cca981204821"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:13.261042 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.261002 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-workload-certs\") pod \"8a9a143a-3fb6-4824-8d4e-cca981204821\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " Nov 03 21:47:13.261311 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.261089 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-data" pod "8a9a143a-3fb6-4824-8d4e-cca981204821" (UID: "8a9a143a-3fb6-4824-8d4e-cca981204821"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:13.261311 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.261151 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-credential-socket\") pod \"8a9a143a-3fb6-4824-8d4e-cca981204821\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " Nov 03 21:47:13.261311 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.261192 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a9a143a-3fb6-4824-8d4e-cca981204821-kube-api-access-wdfsb\" (UniqueName: \"kubernetes.io/projected/8a9a143a-3fb6-4824-8d4e-cca981204821-kube-api-access-wdfsb\") pod \"8a9a143a-3fb6-4824-8d4e-cca981204821\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " Nov 03 21:47:13.261311 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.261273 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-workload-certs" pod "8a9a143a-3fb6-4824-8d4e-cca981204821" (UID: "8a9a143a-3fb6-4824-8d4e-cca981204821"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:13.261532 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.261313 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a9a143a-3fb6-4824-8d4e-cca981204821-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9a143a-3fb6-4824-8d4e-cca981204821-istiod-ca-cert\") pod \"8a9a143a-3fb6-4824-8d4e-cca981204821\" (UID: \"8a9a143a-3fb6-4824-8d4e-cca981204821\") " Nov 03 21:47:13.261532 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.261415 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-credential-socket" pod "8a9a143a-3fb6-4824-8d4e-cca981204821" (UID: "8a9a143a-3fb6-4824-8d4e-cca981204821"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:13.261775 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.261750 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9a143a-3fb6-4824-8d4e-cca981204821-istiod-ca-cert" pod "8a9a143a-3fb6-4824-8d4e-cca981204821" (UID: "8a9a143a-3fb6-4824-8d4e-cca981204821"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:13.261876 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.261861 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.261968 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.261883 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9a143a-3fb6-4824-8d4e-cca981204821-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.261968 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.261933 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.261968 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.261947 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.261968 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.261959 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.263272 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.263246 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-podinfo" pod "8a9a143a-3fb6-4824-8d4e-cca981204821" (UID: "8a9a143a-3fb6-4824-8d4e-cca981204821"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:47:13.263461 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.263442 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-envoy" pod "8a9a143a-3fb6-4824-8d4e-cca981204821" (UID: "8a9a143a-3fb6-4824-8d4e-cca981204821"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:13.263614 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.263599 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-token" pod "8a9a143a-3fb6-4824-8d4e-cca981204821" (UID: "8a9a143a-3fb6-4824-8d4e-cca981204821"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:13.263746 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.263723 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9a143a-3fb6-4824-8d4e-cca981204821-kube-api-access-wdfsb" pod "8a9a143a-3fb6-4824-8d4e-cca981204821" (UID: "8a9a143a-3fb6-4824-8d4e-cca981204821"). InnerVolumeSpecName "kube-api-access-wdfsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:13.363043 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.363004 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdfsb\" (UniqueName: \"kubernetes.io/projected/8a9a143a-3fb6-4824-8d4e-cca981204821-kube-api-access-wdfsb\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.363043 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.363036 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.363043 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.363046 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.363304 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.363057 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9a143a-3fb6-4824-8d4e-cca981204821-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:47:13.684479 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.684437 4963 generic.go:356] "Generic (PLEG): container finished" podID="8a9a143a-3fb6-4824-8d4e-cca981204821" containerID="b2b0cf79e734136647651cd0fac8b70f1514cdbaa16191e61ee9a94db4a9f57e" exitCode=0 Nov 03 21:47:13.684942 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.684490 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" Nov 03 21:47:13.684942 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.684500 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" event={"ID":"8a9a143a-3fb6-4824-8d4e-cca981204821","Type":"ContainerDied","Data":"b2b0cf79e734136647651cd0fac8b70f1514cdbaa16191e61ee9a94db4a9f57e"} Nov 03 21:47:13.684942 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.684550 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk" event={"ID":"8a9a143a-3fb6-4824-8d4e-cca981204821","Type":"ContainerDied","Data":"6583d3a844230abae8bc08220fca1cda7d3b855d2f578c608cda8b73c684716b"} Nov 03 21:47:13.684942 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.684579 4963 scope.go:122] "RemoveContainer" containerID="b2b0cf79e734136647651cd0fac8b70f1514cdbaa16191e61ee9a94db4a9f57e" Nov 03 21:47:13.695678 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.695657 4963 scope.go:122] "RemoveContainer" containerID="b2b0cf79e734136647651cd0fac8b70f1514cdbaa16191e61ee9a94db4a9f57e" Nov 03 21:47:13.696027 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:47:13.695998 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b0cf79e734136647651cd0fac8b70f1514cdbaa16191e61ee9a94db4a9f57e\": container with ID starting with b2b0cf79e734136647651cd0fac8b70f1514cdbaa16191e61ee9a94db4a9f57e not found: ID does not exist" containerID="b2b0cf79e734136647651cd0fac8b70f1514cdbaa16191e61ee9a94db4a9f57e" Nov 03 21:47:13.696131 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.696036 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b0cf79e734136647651cd0fac8b70f1514cdbaa16191e61ee9a94db4a9f57e"} err="failed to get container status \"b2b0cf79e734136647651cd0fac8b70f1514cdbaa16191e61ee9a94db4a9f57e\": rpc error: code = NotFound desc = could not find container \"b2b0cf79e734136647651cd0fac8b70f1514cdbaa16191e61ee9a94db4a9f57e\": container with ID starting with b2b0cf79e734136647651cd0fac8b70f1514cdbaa16191e61ee9a94db4a9f57e not found: ID does not exist" Nov 03 21:47:13.843400 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.843363 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk"] Nov 03 21:47:13.849408 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:13.849380 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-98qrk"] Nov 03 21:47:14.361786 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:14.361741 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:14.557069 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:14.557030 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:14.860855 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:14.860824 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:14.968187 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:14.968151 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a9a143a-3fb6-4824-8d4e-cca981204821" path="/var/lib/kubelet/pods/8a9a143a-3fb6-4824-8d4e-cca981204821/volumes" Nov 03 21:47:15.115718 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:15.115599 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:16.629011 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:16.628977 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:16.877110 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:16.877072 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:18.745660 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:18.745625 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:18.945788 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:18.945763 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:21.202623 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:21.202584 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:21.406590 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:21.406557 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:25.813284 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:25.813250 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:30.417024 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:30.416960 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:31.451436 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:31.451408 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:47:31.466609 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:31.466582 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:47:34.894530 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:34.894494 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:39.417555 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:39.417493 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:41.328394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:41.328328 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:42.528801 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:47:42.528761 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:47:43.918824 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:43.918785 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:48.551613 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:48.551572 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:53.076495 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:53.076457 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:47:57.636311 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:47:57.636275 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:01.566873 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:01.566836 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:03.227889 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:03.227858 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:04.640273 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:04.640233 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:04.827381 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:04.827334 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:06.127085 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:06.127049 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:06.609236 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:06.609206 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:07.385043 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:07.385007 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:07.907080 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:07.907042 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:12.528273 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:48:12.528183 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:48:12.640933 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:12.640872 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:13.216369 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:13.216331 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:13.530213 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:13.530117 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:14.520005 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:14.519963 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j"] Nov 03 21:48:14.941539 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:14.941433 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:15.203197 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:15.203155 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:17.735965 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:17.735925 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:19.144243 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.144205 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc"] Nov 03 21:48:19.144942 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.144793 4963 cpu_manager.go:414] "containerMap: removing container" podUID="16279a5e-a28a-44a6-ba3c-05566f417084" containerName="maas-api" Nov 03 21:48:19.144942 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.144815 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16279a5e-a28a-44a6-ba3c-05566f417084" containerName="maas-api" Nov 03 21:48:19.144942 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.144853 4963 cpu_manager.go:414] "containerMap: removing container" podUID="8a9a143a-3fb6-4824-8d4e-cca981204821" containerName="istio-proxy" Nov 03 21:48:19.144942 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.144860 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a9a143a-3fb6-4824-8d4e-cca981204821" containerName="istio-proxy" Nov 03 21:48:19.144942 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.144886 4963 cpu_manager.go:414] "containerMap: removing container" podUID="a297c4b6-bcfb-4225-9651-5d69ec8b6715" containerName="maas-api" Nov 03 21:48:19.144942 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.144893 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a297c4b6-bcfb-4225-9651-5d69ec8b6715" containerName="maas-api" Nov 03 21:48:19.145342 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.145015 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a9a143a-3fb6-4824-8d4e-cca981204821" containerName="istio-proxy" Nov 03 21:48:19.145342 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.145026 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="a297c4b6-bcfb-4225-9651-5d69ec8b6715" containerName="maas-api" Nov 03 21:48:19.145342 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.145034 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="16279a5e-a28a-44a6-ba3c-05566f417084" containerName="maas-api" Nov 03 21:48:19.149033 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.148978 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.198880 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.198843 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4cjrg\"" Nov 03 21:48:19.211625 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.211593 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.211748 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.211637 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.211748 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.211666 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.211821 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.211759 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.211821 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.211791 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70e4f9f6-f644-40b2-93fb-0acb72d86443-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.211945 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.211933 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fstj6\" (UniqueName: \"kubernetes.io/projected/70e4f9f6-f644-40b2-93fb-0acb72d86443-kube-api-access-fstj6\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.211990 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.211971 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.212026 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.212010 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.212129 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.212105 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.249275 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.249241 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc"] Nov 03 21:48:19.312849 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.312808 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fstj6\" (UniqueName: \"kubernetes.io/projected/70e4f9f6-f644-40b2-93fb-0acb72d86443-kube-api-access-fstj6\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.313145 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.312872 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.313145 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.312959 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.313145 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.313062 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.313145 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.313089 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.313145 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.313130 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.313469 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.313152 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.313469 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.313191 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.313469 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.313229 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70e4f9f6-f644-40b2-93fb-0acb72d86443-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.313469 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.313395 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.313661 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.313522 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.313661 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.313598 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.313869 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.313848 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.314010 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.313995 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70e4f9f6-f644-40b2-93fb-0acb72d86443-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.315410 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.315391 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.315649 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.315630 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.371681 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.371643 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.371862 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.371717 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fstj6\" (UniqueName: \"kubernetes.io/projected/70e4f9f6-f644-40b2-93fb-0acb72d86443-kube-api-access-fstj6\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9n7nc\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.462987 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.462936 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:19.521154 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.521111 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j" podUID="eaebca48-d00b-4f5b-9a25-67f09d3c5f45" containerName="payload-pre-processing" containerID="cri-o://5691e9bafa4a44569631755342f43c916e25458a42e1b5d13ba354c700739da1" gracePeriod=25 Nov 03 21:48:19.774345 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.774315 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j" Nov 03 21:48:19.818769 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.818726 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaebca48-d00b-4f5b-9a25-67f09d3c5f45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaebca48-d00b-4f5b-9a25-67f09d3c5f45-plugins-config-volume\") pod \"eaebca48-d00b-4f5b-9a25-67f09d3c5f45\" (UID: \"eaebca48-d00b-4f5b-9a25-67f09d3c5f45\") " Nov 03 21:48:19.818991 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.818869 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaebca48-d00b-4f5b-9a25-67f09d3c5f45-kube-api-access-r6dx7\" (UniqueName: \"kubernetes.io/projected/eaebca48-d00b-4f5b-9a25-67f09d3c5f45-kube-api-access-r6dx7\") pod \"eaebca48-d00b-4f5b-9a25-67f09d3c5f45\" (UID: \"eaebca48-d00b-4f5b-9a25-67f09d3c5f45\") " Nov 03 21:48:19.819085 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.819063 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaebca48-d00b-4f5b-9a25-67f09d3c5f45-plugins-config-volume" pod "eaebca48-d00b-4f5b-9a25-67f09d3c5f45" (UID: "eaebca48-d00b-4f5b-9a25-67f09d3c5f45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:19.819299 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.819284 4963 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaebca48-d00b-4f5b-9a25-67f09d3c5f45-plugins-config-volume\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:19.821330 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.821296 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaebca48-d00b-4f5b-9a25-67f09d3c5f45-kube-api-access-r6dx7" pod "eaebca48-d00b-4f5b-9a25-67f09d3c5f45" (UID: "eaebca48-d00b-4f5b-9a25-67f09d3c5f45"). InnerVolumeSpecName "kube-api-access-r6dx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:19.896971 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.896930 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc"] Nov 03 21:48:19.901838 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:48:19.901800 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e4f9f6_f644_40b2_93fb_0acb72d86443.slice/crio-237ad02c8857f956664667a5d15bf07e65a52ff6b274d1800bf1a113e70604be WatchSource:0}: Error finding container 237ad02c8857f956664667a5d15bf07e65a52ff6b274d1800bf1a113e70604be: Status 404 returned error can't find the container with id 237ad02c8857f956664667a5d15bf07e65a52ff6b274d1800bf1a113e70604be Nov 03 21:48:19.905003 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.904967 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:48:19.905114 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.905041 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:48:19.905114 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.905069 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:48:19.916526 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.916494 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:19.920275 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:19.920245 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6dx7\" (UniqueName: \"kubernetes.io/projected/eaebca48-d00b-4f5b-9a25-67f09d3c5f45-kube-api-access-r6dx7\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:20.103178 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:20.103093 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:20.125109 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:20.125075 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" event={"ID":"70e4f9f6-f644-40b2-93fb-0acb72d86443","Type":"ContainerStarted","Data":"a51f11f7e0c775a7b707c28ab576b0744ef3d221aec9b9c3afacadefd83e90ff"} Nov 03 21:48:20.125298 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:20.125130 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" event={"ID":"70e4f9f6-f644-40b2-93fb-0acb72d86443","Type":"ContainerStarted","Data":"237ad02c8857f956664667a5d15bf07e65a52ff6b274d1800bf1a113e70604be"} Nov 03 21:48:20.126363 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:20.126337 4963 generic.go:356] "Generic (PLEG): container finished" podID="eaebca48-d00b-4f5b-9a25-67f09d3c5f45" containerID="5691e9bafa4a44569631755342f43c916e25458a42e1b5d13ba354c700739da1" exitCode=0 Nov 03 21:48:20.126495 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:20.126367 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j" Nov 03 21:48:20.126495 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:20.126369 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j" event={"ID":"eaebca48-d00b-4f5b-9a25-67f09d3c5f45","Type":"ContainerDied","Data":"5691e9bafa4a44569631755342f43c916e25458a42e1b5d13ba354c700739da1"} Nov 03 21:48:20.126613 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:20.126502 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j" event={"ID":"eaebca48-d00b-4f5b-9a25-67f09d3c5f45","Type":"ContainerDied","Data":"3b098c59a4107f46f0df5f921692a715b7240c4ab6dba322ac60e05314c84d78"} Nov 03 21:48:20.126613 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:20.126537 4963 scope.go:122] "RemoveContainer" containerID="5691e9bafa4a44569631755342f43c916e25458a42e1b5d13ba354c700739da1" Nov 03 21:48:20.138070 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:20.138042 4963 scope.go:122] "RemoveContainer" containerID="5691e9bafa4a44569631755342f43c916e25458a42e1b5d13ba354c700739da1" Nov 03 21:48:20.138435 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:48:20.138414 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5691e9bafa4a44569631755342f43c916e25458a42e1b5d13ba354c700739da1\": container with ID starting with 5691e9bafa4a44569631755342f43c916e25458a42e1b5d13ba354c700739da1 not found: ID does not exist" containerID="5691e9bafa4a44569631755342f43c916e25458a42e1b5d13ba354c700739da1" Nov 03 21:48:20.138502 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:20.138449 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5691e9bafa4a44569631755342f43c916e25458a42e1b5d13ba354c700739da1"} err="failed to get container status \"5691e9bafa4a44569631755342f43c916e25458a42e1b5d13ba354c700739da1\": rpc error: code = NotFound desc = could not find container \"5691e9bafa4a44569631755342f43c916e25458a42e1b5d13ba354c700739da1\": container with ID starting with 5691e9bafa4a44569631755342f43c916e25458a42e1b5d13ba354c700739da1 not found: ID does not exist" Nov 03 21:48:20.244186 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:20.244123 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" podStartSLOduration=1.2441005330000001 podStartE2EDuration="1.244100533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:20.239783405 +0000 UTC m=+3649.399855679" watchObservedRunningTime="2026-11-03 21:48:20.244100533 +0000 UTC m=+3649.404172807" Nov 03 21:48:20.286015 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:20.285978 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:20.347806 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:20.347765 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j"] Nov 03 21:48:20.353835 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:20.353767 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a02404-78fdb4d97f-6nd9j"] Nov 03 21:48:20.463955 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:20.463889 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:20.465806 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:20.465774 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 21:48:20.465962 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:20.465834 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" podUID="70e4f9f6-f644-40b2-93fb-0acb72d86443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 21:48:20.968448 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:20.968413 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaebca48-d00b-4f5b-9a25-67f09d3c5f45" path="/var/lib/kubelet/pods/eaebca48-d00b-4f5b-9a25-67f09d3c5f45/volumes" Nov 03 21:48:21.463642 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:21.463597 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 21:48:21.464129 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:21.463658 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" podUID="70e4f9f6-f644-40b2-93fb-0acb72d86443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 21:48:22.464006 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:22.463969 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 21:48:22.464512 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:22.464046 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" podUID="70e4f9f6-f644-40b2-93fb-0acb72d86443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 21:48:22.574114 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:22.574075 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:23.464304 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:23.464271 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 21:48:23.464821 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:23.464320 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" podUID="70e4f9f6-f644-40b2-93fb-0acb72d86443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 21:48:24.463868 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:24.463829 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 21:48:24.464073 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:24.463886 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" podUID="70e4f9f6-f644-40b2-93fb-0acb72d86443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 21:48:25.463475 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:25.463434 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 21:48:25.463882 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:25.463500 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" podUID="70e4f9f6-f644-40b2-93fb-0acb72d86443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 21:48:25.806056 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:25.802574 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:26.463821 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:26.463782 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 21:48:26.464301 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:26.463848 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" podUID="70e4f9f6-f644-40b2-93fb-0acb72d86443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 21:48:27.157366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:27.157266 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:27.463888 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:27.463849 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 21:48:27.464381 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:27.463921 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" podUID="70e4f9f6-f644-40b2-93fb-0acb72d86443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 21:48:28.464008 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:28.463968 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 21:48:28.464551 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:28.464034 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" podUID="70e4f9f6-f644-40b2-93fb-0acb72d86443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 21:48:29.636849 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:29.636812 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:29.637703 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:29.637497 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:29.638561 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:29.638538 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 21:48:31.798632 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:31.798582 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:36.027148 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.027108 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75"] Nov 03 21:48:36.027706 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.027690 4963 cpu_manager.go:414] "containerMap: removing container" podUID="eaebca48-d00b-4f5b-9a25-67f09d3c5f45" containerName="payload-pre-processing" Nov 03 21:48:36.027753 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.027707 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaebca48-d00b-4f5b-9a25-67f09d3c5f45" containerName="payload-pre-processing" Nov 03 21:48:36.027837 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.027828 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaebca48-d00b-4f5b-9a25-67f09d3c5f45" containerName="payload-pre-processing" Nov 03 21:48:36.031135 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.031114 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:48:36.128130 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.128087 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-267053-serving-cert\"" Nov 03 21:48:36.131347 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.131317 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75"] Nov 03 21:48:36.156555 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.156507 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15848a43-dcff-43db-801c-4ca6830d16d8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-267053-5599d7f969-wzz75\" (UID: \"15848a43-dcff-43db-801c-4ca6830d16d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:48:36.156821 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.156786 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15848a43-dcff-43db-801c-4ca6830d16d8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-267053-5599d7f969-wzz75\" (UID: \"15848a43-dcff-43db-801c-4ca6830d16d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:48:36.157018 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.156847 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15848a43-dcff-43db-801c-4ca6830d16d8-metrics-certs\") pod \"maas-api-e2e-extmod-a-267053-5599d7f969-wzz75\" (UID: \"15848a43-dcff-43db-801c-4ca6830d16d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:48:36.157106 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.157030 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6drd8\" (UniqueName: \"kubernetes.io/projected/15848a43-dcff-43db-801c-4ca6830d16d8-kube-api-access-6drd8\") pod \"maas-api-e2e-extmod-a-267053-5599d7f969-wzz75\" (UID: \"15848a43-dcff-43db-801c-4ca6830d16d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:48:36.258718 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.258676 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15848a43-dcff-43db-801c-4ca6830d16d8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-267053-5599d7f969-wzz75\" (UID: \"15848a43-dcff-43db-801c-4ca6830d16d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:48:36.258973 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.258735 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15848a43-dcff-43db-801c-4ca6830d16d8-metrics-certs\") pod \"maas-api-e2e-extmod-a-267053-5599d7f969-wzz75\" (UID: \"15848a43-dcff-43db-801c-4ca6830d16d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:48:36.258973 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.258790 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6drd8\" (UniqueName: \"kubernetes.io/projected/15848a43-dcff-43db-801c-4ca6830d16d8-kube-api-access-6drd8\") pod \"maas-api-e2e-extmod-a-267053-5599d7f969-wzz75\" (UID: \"15848a43-dcff-43db-801c-4ca6830d16d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:48:36.258973 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.258827 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15848a43-dcff-43db-801c-4ca6830d16d8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-267053-5599d7f969-wzz75\" (UID: \"15848a43-dcff-43db-801c-4ca6830d16d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:48:36.258973 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:48:36.258837 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-serving-cert: secret "maas-api-e2e-extmod-a-267053-serving-cert" not found Nov 03 21:48:36.258973 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:48:36.258941 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15848a43-dcff-43db-801c-4ca6830d16d8-maas-api-tls podName:15848a43-dcff-43db-801c-4ca6830d16d8 nodeName:}" failed. No retries permitted until 2026-11-03 21:48:36.758918817 +0000 UTC m=+3665.918991069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15848a43-dcff-43db-801c-4ca6830d16d8-maas-api-tls") pod "maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" (UID: "15848a43-dcff-43db-801c-4ca6830d16d8") : secret "maas-api-e2e-extmod-a-267053-serving-cert" not found Nov 03 21:48:36.259573 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.259547 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15848a43-dcff-43db-801c-4ca6830d16d8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-267053-5599d7f969-wzz75\" (UID: \"15848a43-dcff-43db-801c-4ca6830d16d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:48:36.259702 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.259549 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:36.261512 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.261492 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15848a43-dcff-43db-801c-4ca6830d16d8-metrics-certs\") pod \"maas-api-e2e-extmod-a-267053-5599d7f969-wzz75\" (UID: \"15848a43-dcff-43db-801c-4ca6830d16d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:48:36.318373 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.318298 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6drd8\" (UniqueName: \"kubernetes.io/projected/15848a43-dcff-43db-801c-4ca6830d16d8-kube-api-access-6drd8\") pod \"maas-api-e2e-extmod-a-267053-5599d7f969-wzz75\" (UID: \"15848a43-dcff-43db-801c-4ca6830d16d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:48:36.441242 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.441205 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:36.620887 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.620806 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:36.764674 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.764626 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15848a43-dcff-43db-801c-4ca6830d16d8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-267053-5599d7f969-wzz75\" (UID: \"15848a43-dcff-43db-801c-4ca6830d16d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:48:36.767163 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.767141 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15848a43-dcff-43db-801c-4ca6830d16d8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-267053-5599d7f969-wzz75\" (UID: \"15848a43-dcff-43db-801c-4ca6830d16d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:48:36.945978 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:36.945942 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:48:37.322445 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:37.322358 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75"] Nov 03 21:48:37.327622 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:48:37.327591 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15848a43_dcff_43db_801c_4ca6830d16d8.slice/crio-2575426bf6cac9184d674d87fafb8da4b326cf4ac6728049c142c45336e6688b WatchSource:0}: Error finding container 2575426bf6cac9184d674d87fafb8da4b326cf4ac6728049c142c45336e6688b: Status 404 returned error can't find the container with id 2575426bf6cac9184d674d87fafb8da4b326cf4ac6728049c142c45336e6688b Nov 03 21:48:38.259498 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:38.259458 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" event={"ID":"15848a43-dcff-43db-801c-4ca6830d16d8","Type":"ContainerStarted","Data":"2575426bf6cac9184d674d87fafb8da4b326cf4ac6728049c142c45336e6688b"} Nov 03 21:48:39.267419 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:39.267371 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" event={"ID":"15848a43-dcff-43db-801c-4ca6830d16d8","Type":"ContainerStarted","Data":"67a35bb03a26c9bb0d82912810640c99560f7193effffe602699fd2ac5227a23"} Nov 03 21:48:39.267832 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:39.267457 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:48:39.381008 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:39.380951 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" podStartSLOduration=3.500780636 podStartE2EDuration="4.380937231s" totalImagesPullingTime="880.156595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:39.378565073 +0000 UTC m=+3668.538637348" watchObservedRunningTime="2026-11-03 21:48:39.380937231 +0000 UTC m=+3668.541009503" Nov 03 21:48:40.209788 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:40.209749 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf"] Nov 03 21:48:40.210025 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:40.209965 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" podUID="2b1f5ae3-b769-4b92-8f54-f45c2393a4e7" containerName="maas-api" containerID="cri-o://2e0f301273b8a2b633d82e7265b5638a0081ee11ae63ef00adc57d5e59f08779" gracePeriod=30 Nov 03 21:48:40.347603 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:40.347563 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:40.538963 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:40.538934 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" Nov 03 21:48:40.716374 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:40.716339 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-metrics-certs\") pod \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\" (UID: \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\") " Nov 03 21:48:40.716586 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:40.716396 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-kube-api-access-vl8kk\" (UniqueName: \"kubernetes.io/projected/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-kube-api-access-vl8kk\") pod \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\" (UID: \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\") " Nov 03 21:48:40.716586 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:40.716445 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-openshift-service-ca\") pod \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\" (UID: \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\") " Nov 03 21:48:40.716586 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:40.716493 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-maas-api-tls\") pod \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\" (UID: \"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7\") " Nov 03 21:48:40.716871 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:40.716841 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-openshift-service-ca" pod "2b1f5ae3-b769-4b92-8f54-f45c2393a4e7" (UID: "2b1f5ae3-b769-4b92-8f54-f45c2393a4e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:40.717051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:40.717020 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:40.718741 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:40.718716 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-maas-api-tls" pod "2b1f5ae3-b769-4b92-8f54-f45c2393a4e7" (UID: "2b1f5ae3-b769-4b92-8f54-f45c2393a4e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:48:40.718848 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:40.718763 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-metrics-certs" pod "2b1f5ae3-b769-4b92-8f54-f45c2393a4e7" (UID: "2b1f5ae3-b769-4b92-8f54-f45c2393a4e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:48:40.718848 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:40.718836 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-kube-api-access-vl8kk" pod "2b1f5ae3-b769-4b92-8f54-f45c2393a4e7" (UID: "2b1f5ae3-b769-4b92-8f54-f45c2393a4e7"). InnerVolumeSpecName "kube-api-access-vl8kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:40.818605 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:40.818514 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:40.818605 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:40.818550 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl8kk\" (UniqueName: \"kubernetes.io/projected/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-kube-api-access-vl8kk\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:40.818605 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:40.818561 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:48:41.284617 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:41.284575 4963 generic.go:356] "Generic (PLEG): container finished" podID="2b1f5ae3-b769-4b92-8f54-f45c2393a4e7" containerID="2e0f301273b8a2b633d82e7265b5638a0081ee11ae63ef00adc57d5e59f08779" exitCode=0 Nov 03 21:48:41.284813 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:41.284644 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" Nov 03 21:48:41.284813 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:41.284675 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" event={"ID":"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7","Type":"ContainerDied","Data":"2e0f301273b8a2b633d82e7265b5638a0081ee11ae63ef00adc57d5e59f08779"} Nov 03 21:48:41.284813 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:41.284740 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf" event={"ID":"2b1f5ae3-b769-4b92-8f54-f45c2393a4e7","Type":"ContainerDied","Data":"a3b3f8ceaae1820f9c0dde9de520405ebbb0944fdf1d46c8614096609e78eacd"} Nov 03 21:48:41.284813 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:41.284770 4963 scope.go:122] "RemoveContainer" containerID="2e0f301273b8a2b633d82e7265b5638a0081ee11ae63ef00adc57d5e59f08779" Nov 03 21:48:41.296226 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:41.296198 4963 scope.go:122] "RemoveContainer" containerID="2e0f301273b8a2b633d82e7265b5638a0081ee11ae63ef00adc57d5e59f08779" Nov 03 21:48:41.296575 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:48:41.296548 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0f301273b8a2b633d82e7265b5638a0081ee11ae63ef00adc57d5e59f08779\": container with ID starting with 2e0f301273b8a2b633d82e7265b5638a0081ee11ae63ef00adc57d5e59f08779 not found: ID does not exist" containerID="2e0f301273b8a2b633d82e7265b5638a0081ee11ae63ef00adc57d5e59f08779" Nov 03 21:48:41.296647 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:41.296587 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0f301273b8a2b633d82e7265b5638a0081ee11ae63ef00adc57d5e59f08779"} err="failed to get container status \"2e0f301273b8a2b633d82e7265b5638a0081ee11ae63ef00adc57d5e59f08779\": rpc error: code = NotFound desc = could not find container \"2e0f301273b8a2b633d82e7265b5638a0081ee11ae63ef00adc57d5e59f08779\": container with ID starting with 2e0f301273b8a2b633d82e7265b5638a0081ee11ae63ef00adc57d5e59f08779 not found: ID does not exist" Nov 03 21:48:41.442511 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:41.442478 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf"] Nov 03 21:48:41.489978 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:41.489943 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0fc5f-9cdf5f697-ztsnf"] Nov 03 21:48:41.674768 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:41.674690 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:41.949692 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:41.949664 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:42.228711 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:42.228634 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:42.518144 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:42.518059 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:42.529248 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:48:42.529207 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:48:42.968007 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:42.967976 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b1f5ae3-b769-4b92-8f54-f45c2393a4e7" path="/var/lib/kubelet/pods/2b1f5ae3-b769-4b92-8f54-f45c2393a4e7/volumes" Nov 03 21:48:45.283602 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:45.283570 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:48:45.704016 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:45.703971 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:45.888749 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:45.888710 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:46.834289 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:46.834249 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:49.578664 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:49.578627 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:49.771915 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:49.771870 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:51.451326 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:51.451298 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:53.210139 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:53.210106 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:53.405299 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:53.405250 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:54.974161 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:54.974120 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:55.157503 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:55.157459 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:55.895069 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:55.895036 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:56.281531 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:56.281465 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:56.884403 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:56.884372 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:57.466505 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:57.466470 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:57.652480 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:57.652429 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:57.842670 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:57.842542 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:48:58.189768 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:48:58.189672 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:00.211508 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:00.211475 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:00.571291 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:00.571166 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:01.580643 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:01.580607 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:01.771311 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:01.771280 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:03.600104 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:03.600071 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:03.780774 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:03.780738 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:05.619172 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:05.619126 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:05.806234 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:05.806189 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:07.010173 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:07.010126 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:09.843764 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:09.843725 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:10.026478 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:10.026447 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:10.416152 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:10.416117 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:12.535716 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:49:12.535674 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:49:13.409861 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:13.409804 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:15.045370 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:15.045330 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:17.115597 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:17.115557 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:17.312368 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:17.312336 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:17.613163 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:17.613121 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:18.859875 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:18.859836 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:19.461853 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:19.461820 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:19.659256 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:19.659219 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:19.854765 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:19.854678 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:23.941162 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:23.941124 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:24.279618 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:24.279584 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:26.048209 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:26.048173 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:27.252520 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:27.252481 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:27.453045 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:27.453002 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:28.290948 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.290899 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6"] Nov 03 21:49:28.291350 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.291127 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" podUID="34356d65-dc53-4d4c-bffe-341dbd8653b3" containerName="maas-api" containerID="cri-o://611b51a8c4a588430ac0719d50c0ac9929f9fb786a157dea9cf44caed8b0d0b7" gracePeriod=30 Nov 03 21:49:28.428938 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.428510 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:28.588674 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.588637 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:49:28.633556 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.633522 4963 generic.go:356] "Generic (PLEG): container finished" podID="34356d65-dc53-4d4c-bffe-341dbd8653b3" containerID="611b51a8c4a588430ac0719d50c0ac9929f9fb786a157dea9cf44caed8b0d0b7" exitCode=0 Nov 03 21:49:28.633758 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.633575 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" Nov 03 21:49:28.633758 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.633567 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" event={"ID":"34356d65-dc53-4d4c-bffe-341dbd8653b3","Type":"ContainerDied","Data":"611b51a8c4a588430ac0719d50c0ac9929f9fb786a157dea9cf44caed8b0d0b7"} Nov 03 21:49:28.633758 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.633630 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6" event={"ID":"34356d65-dc53-4d4c-bffe-341dbd8653b3","Type":"ContainerDied","Data":"c07c0162f9128a75a824cd802dd6da45859fcc6b11f6eb496dc6d4ee25c1857d"} Nov 03 21:49:28.633758 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.633652 4963 scope.go:122] "RemoveContainer" containerID="611b51a8c4a588430ac0719d50c0ac9929f9fb786a157dea9cf44caed8b0d0b7" Nov 03 21:49:28.644642 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.644608 4963 scope.go:122] "RemoveContainer" containerID="611b51a8c4a588430ac0719d50c0ac9929f9fb786a157dea9cf44caed8b0d0b7" Nov 03 21:49:28.645052 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:49:28.645025 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611b51a8c4a588430ac0719d50c0ac9929f9fb786a157dea9cf44caed8b0d0b7\": container with ID starting with 611b51a8c4a588430ac0719d50c0ac9929f9fb786a157dea9cf44caed8b0d0b7 not found: ID does not exist" containerID="611b51a8c4a588430ac0719d50c0ac9929f9fb786a157dea9cf44caed8b0d0b7" Nov 03 21:49:28.645168 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.645064 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611b51a8c4a588430ac0719d50c0ac9929f9fb786a157dea9cf44caed8b0d0b7"} err="failed to get container status \"611b51a8c4a588430ac0719d50c0ac9929f9fb786a157dea9cf44caed8b0d0b7\": rpc error: code = NotFound desc = could not find container \"611b51a8c4a588430ac0719d50c0ac9929f9fb786a157dea9cf44caed8b0d0b7\": container with ID starting with 611b51a8c4a588430ac0719d50c0ac9929f9fb786a157dea9cf44caed8b0d0b7 not found: ID does not exist" Nov 03 21:49:28.693893 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.693856 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34356d65-dc53-4d4c-bffe-341dbd8653b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34356d65-dc53-4d4c-bffe-341dbd8653b3-openshift-service-ca\") pod \"34356d65-dc53-4d4c-bffe-341dbd8653b3\" (UID: \"34356d65-dc53-4d4c-bffe-341dbd8653b3\") " Nov 03 21:49:28.694101 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.693949 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34356d65-dc53-4d4c-bffe-341dbd8653b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34356d65-dc53-4d4c-bffe-341dbd8653b3-metrics-certs\") pod \"34356d65-dc53-4d4c-bffe-341dbd8653b3\" (UID: \"34356d65-dc53-4d4c-bffe-341dbd8653b3\") " Nov 03 21:49:28.694101 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.694022 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34356d65-dc53-4d4c-bffe-341dbd8653b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34356d65-dc53-4d4c-bffe-341dbd8653b3-maas-api-tls\") pod \"34356d65-dc53-4d4c-bffe-341dbd8653b3\" (UID: \"34356d65-dc53-4d4c-bffe-341dbd8653b3\") " Nov 03 21:49:28.694101 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.694093 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34356d65-dc53-4d4c-bffe-341dbd8653b3-kube-api-access-bh5v7\" (UniqueName: \"kubernetes.io/projected/34356d65-dc53-4d4c-bffe-341dbd8653b3-kube-api-access-bh5v7\") pod \"34356d65-dc53-4d4c-bffe-341dbd8653b3\" (UID: \"34356d65-dc53-4d4c-bffe-341dbd8653b3\") " Nov 03 21:49:28.694311 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.694285 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34356d65-dc53-4d4c-bffe-341dbd8653b3-openshift-service-ca" pod "34356d65-dc53-4d4c-bffe-341dbd8653b3" (UID: "34356d65-dc53-4d4c-bffe-341dbd8653b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:49:28.694489 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.694470 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34356d65-dc53-4d4c-bffe-341dbd8653b3-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:28.696393 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.696367 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34356d65-dc53-4d4c-bffe-341dbd8653b3-metrics-certs" pod "34356d65-dc53-4d4c-bffe-341dbd8653b3" (UID: "34356d65-dc53-4d4c-bffe-341dbd8653b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:49:28.696393 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.696384 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34356d65-dc53-4d4c-bffe-341dbd8653b3-maas-api-tls" pod "34356d65-dc53-4d4c-bffe-341dbd8653b3" (UID: "34356d65-dc53-4d4c-bffe-341dbd8653b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:49:28.696568 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.696407 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34356d65-dc53-4d4c-bffe-341dbd8653b3-kube-api-access-bh5v7" pod "34356d65-dc53-4d4c-bffe-341dbd8653b3" (UID: "34356d65-dc53-4d4c-bffe-341dbd8653b3"). InnerVolumeSpecName "kube-api-access-bh5v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:28.795557 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.795517 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34356d65-dc53-4d4c-bffe-341dbd8653b3-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:28.795557 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.795550 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34356d65-dc53-4d4c-bffe-341dbd8653b3-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:28.795557 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:28.795561 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh5v7\" (UniqueName: \"kubernetes.io/projected/34356d65-dc53-4d4c-bffe-341dbd8653b3-kube-api-access-bh5v7\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:29.060085 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.060018 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6"] Nov 03 21:49:29.068608 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.068579 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2057ca-847c757f64-vdnq6"] Nov 03 21:49:29.418537 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.418453 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns"] Nov 03 21:49:29.418934 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.418664 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" podUID="d504c953-1e7c-4605-a17b-fa5fa0dabed3" containerName="maas-api" containerID="cri-o://2f25ee7b37acf68f6e0fe4be6ee678ac0f42c587423383bd13f5afc8f0553a2b" gracePeriod=30 Nov 03 21:49:29.559220 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.559185 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:29.658138 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.658101 4963 generic.go:356] "Generic (PLEG): container finished" podID="d504c953-1e7c-4605-a17b-fa5fa0dabed3" containerID="2f25ee7b37acf68f6e0fe4be6ee678ac0f42c587423383bd13f5afc8f0553a2b" exitCode=0 Nov 03 21:49:29.658311 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.658167 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" event={"ID":"d504c953-1e7c-4605-a17b-fa5fa0dabed3","Type":"ContainerDied","Data":"2f25ee7b37acf68f6e0fe4be6ee678ac0f42c587423383bd13f5afc8f0553a2b"} Nov 03 21:49:29.722182 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.722146 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:49:29.806528 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.806492 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d504c953-1e7c-4605-a17b-fa5fa0dabed3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d504c953-1e7c-4605-a17b-fa5fa0dabed3-maas-api-tls\") pod \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\" (UID: \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\") " Nov 03 21:49:29.806743 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.806630 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d504c953-1e7c-4605-a17b-fa5fa0dabed3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d504c953-1e7c-4605-a17b-fa5fa0dabed3-metrics-certs\") pod \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\" (UID: \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\") " Nov 03 21:49:29.806743 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.806702 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d504c953-1e7c-4605-a17b-fa5fa0dabed3-kube-api-access-65xzb\" (UniqueName: \"kubernetes.io/projected/d504c953-1e7c-4605-a17b-fa5fa0dabed3-kube-api-access-65xzb\") pod \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\" (UID: \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\") " Nov 03 21:49:29.806743 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.806721 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d504c953-1e7c-4605-a17b-fa5fa0dabed3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d504c953-1e7c-4605-a17b-fa5fa0dabed3-openshift-service-ca\") pod \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\" (UID: \"d504c953-1e7c-4605-a17b-fa5fa0dabed3\") " Nov 03 21:49:29.807186 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.807159 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d504c953-1e7c-4605-a17b-fa5fa0dabed3-openshift-service-ca" pod "d504c953-1e7c-4605-a17b-fa5fa0dabed3" (UID: "d504c953-1e7c-4605-a17b-fa5fa0dabed3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:49:29.809076 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.809048 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d504c953-1e7c-4605-a17b-fa5fa0dabed3-kube-api-access-65xzb" pod "d504c953-1e7c-4605-a17b-fa5fa0dabed3" (UID: "d504c953-1e7c-4605-a17b-fa5fa0dabed3"). InnerVolumeSpecName "kube-api-access-65xzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:29.809223 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.809207 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d504c953-1e7c-4605-a17b-fa5fa0dabed3-maas-api-tls" pod "d504c953-1e7c-4605-a17b-fa5fa0dabed3" (UID: "d504c953-1e7c-4605-a17b-fa5fa0dabed3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:49:29.809392 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.809358 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d504c953-1e7c-4605-a17b-fa5fa0dabed3-metrics-certs" pod "d504c953-1e7c-4605-a17b-fa5fa0dabed3" (UID: "d504c953-1e7c-4605-a17b-fa5fa0dabed3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:49:29.907728 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.907686 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65xzb\" (UniqueName: \"kubernetes.io/projected/d504c953-1e7c-4605-a17b-fa5fa0dabed3-kube-api-access-65xzb\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:29.907728 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.907726 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d504c953-1e7c-4605-a17b-fa5fa0dabed3-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:29.907728 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.907737 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d504c953-1e7c-4605-a17b-fa5fa0dabed3-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:29.908009 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:29.907746 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d504c953-1e7c-4605-a17b-fa5fa0dabed3-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:30.668135 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:30.668094 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" event={"ID":"d504c953-1e7c-4605-a17b-fa5fa0dabed3","Type":"ContainerDied","Data":"ef2a8451cee95d5800f60f78f58c609a9e020e2f33876290353dcbdf57348bd8"} Nov 03 21:49:30.668567 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:30.668177 4963 scope.go:122] "RemoveContainer" containerID="2f25ee7b37acf68f6e0fe4be6ee678ac0f42c587423383bd13f5afc8f0553a2b" Nov 03 21:49:30.673434 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:30.668974 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns" Nov 03 21:49:30.840411 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:30.840374 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns"] Nov 03 21:49:30.892165 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:30.892129 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02404-6c9cb659c4-zm5ns"] Nov 03 21:49:30.971070 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:30.971036 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34356d65-dc53-4d4c-bffe-341dbd8653b3" path="/var/lib/kubelet/pods/34356d65-dc53-4d4c-bffe-341dbd8653b3/volumes" Nov 03 21:49:30.971523 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:30.971508 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d504c953-1e7c-4605-a17b-fa5fa0dabed3" path="/var/lib/kubelet/pods/d504c953-1e7c-4605-a17b-fa5fa0dabed3/volumes" Nov 03 21:49:31.351332 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:31.351241 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:34.738121 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.738071 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9"] Nov 03 21:49:34.738631 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.738617 4963 cpu_manager.go:414] "containerMap: removing container" podUID="2b1f5ae3-b769-4b92-8f54-f45c2393a4e7" containerName="maas-api" Nov 03 21:49:34.738678 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.738632 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b1f5ae3-b769-4b92-8f54-f45c2393a4e7" containerName="maas-api" Nov 03 21:49:34.738678 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.738664 4963 cpu_manager.go:414] "containerMap: removing container" podUID="d504c953-1e7c-4605-a17b-fa5fa0dabed3" containerName="maas-api" Nov 03 21:49:34.738678 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.738669 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d504c953-1e7c-4605-a17b-fa5fa0dabed3" containerName="maas-api" Nov 03 21:49:34.738775 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.738682 4963 cpu_manager.go:414] "containerMap: removing container" podUID="34356d65-dc53-4d4c-bffe-341dbd8653b3" containerName="maas-api" Nov 03 21:49:34.738775 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.738688 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34356d65-dc53-4d4c-bffe-341dbd8653b3" containerName="maas-api" Nov 03 21:49:34.738775 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.738771 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="d504c953-1e7c-4605-a17b-fa5fa0dabed3" containerName="maas-api" Nov 03 21:49:34.738870 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.738781 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="34356d65-dc53-4d4c-bffe-341dbd8653b3" containerName="maas-api" Nov 03 21:49:34.738870 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.738790 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b1f5ae3-b769-4b92-8f54-f45c2393a4e7" containerName="maas-api" Nov 03 21:49:34.742382 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.742362 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.864825 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.864792 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.865035 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.864831 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.865035 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.864953 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.865128 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.865046 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.865128 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.865092 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.865230 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.865214 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.865307 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.865293 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmgb6\" (UniqueName: \"kubernetes.io/projected/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-kube-api-access-gmgb6\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.865343 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.865319 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.865343 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.865338 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.944937 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.944890 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9"] Nov 03 21:49:34.966701 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.966669 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.966940 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.966720 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmgb6\" (UniqueName: \"kubernetes.io/projected/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-kube-api-access-gmgb6\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.966940 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.966741 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.966940 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.966796 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.966940 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.966874 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.966940 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.966916 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.967254 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.966978 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.967254 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.967023 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.967254 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.967053 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.967254 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.967150 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.967475 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.967406 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.967535 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.967499 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.967665 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.967637 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.967665 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.967656 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.969650 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.969628 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:34.970169 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:34.970152 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:35.030482 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:35.030399 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:35.030658 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:35.030632 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmgb6\" (UniqueName: \"kubernetes.io/projected/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-kube-api-access-gmgb6\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-vlgf9\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:35.056664 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:35.056631 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:35.469568 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:35.469529 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:35.492042 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:49:35.492012 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ade2d0_82c2_4a55_9bfa_d6f5916c81fd.slice/crio-1097962693b2e429e2181aa339a856c3ec4776a6e7ce227f07a32556c13fa5e9 WatchSource:0}: Error finding container 1097962693b2e429e2181aa339a856c3ec4776a6e7ce227f07a32556c13fa5e9: Status 404 returned error can't find the container with id 1097962693b2e429e2181aa339a856c3ec4776a6e7ce227f07a32556c13fa5e9 Nov 03 21:49:35.492390 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:35.492368 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9"] Nov 03 21:49:35.494606 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:35.494578 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:49:35.494677 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:35.494638 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:49:35.494714 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:35.494664 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:49:35.653841 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:35.653813 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:35.703157 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:35.703122 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" event={"ID":"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd","Type":"ContainerStarted","Data":"9136ae17435fb96c295861070551ecec36b61b2340cdc1e5289ac35928deb1a9"} Nov 03 21:49:35.703345 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:35.703180 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" event={"ID":"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd","Type":"ContainerStarted","Data":"1097962693b2e429e2181aa339a856c3ec4776a6e7ce227f07a32556c13fa5e9"} Nov 03 21:49:35.829612 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:35.829507 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" podStartSLOduration=1.8294804930000002 podStartE2EDuration="1.829480493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:35.827229185 +0000 UTC m=+3724.987301458" watchObservedRunningTime="2026-11-03 21:49:35.829480493 +0000 UTC m=+3724.989552764" Nov 03 21:49:36.057216 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:36.057171 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:36.058945 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:36.058890 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 21:49:36.059108 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:36.058977 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" podUID="98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 21:49:36.205967 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:36.205930 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:37.057818 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:37.057779 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 21:49:37.058276 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:37.057832 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" podUID="98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 21:49:37.526497 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:37.526457 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:37.807752 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:37.807657 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:38.013614 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:38.013568 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:38.057227 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:38.057190 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 21:49:38.057415 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:38.057245 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" podUID="98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 21:49:38.207358 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:38.207313 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:38.844509 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:38.844462 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:39.031224 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:39.031175 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:39.057353 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:39.057313 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 21:49:39.057528 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:39.057375 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" podUID="98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 21:49:39.360830 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:39.360796 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:40.057187 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:40.057098 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 21:49:40.057187 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:40.057163 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" podUID="98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 21:49:40.254798 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:40.254757 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:40.889027 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:40.888993 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:41.058128 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:41.058092 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 21:49:41.058294 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:41.058161 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" podUID="98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 21:49:41.258620 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:41.258584 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:41.444037 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:41.444005 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:42.020999 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:42.020956 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:42.057479 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:42.057440 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 21:49:42.057707 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:42.057531 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" podUID="98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 21:49:42.536995 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:49:42.536952 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:49:43.057610 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:43.057568 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 21:49:43.058064 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:43.057642 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" podUID="98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 21:49:44.206709 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:44.206675 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:44.207263 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:44.207144 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:44.207954 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:44.207934 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 21:49:44.620713 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:44.620601 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:44.830769 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:44.830734 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:45.667548 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:45.667518 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:46.285963 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:46.285899 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:46.383818 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:46.383782 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd"] Nov 03 21:49:46.680935 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:46.680837 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:46.872718 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:46.872680 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:49.865059 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:49.865009 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6"] Nov 03 21:49:49.870006 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:49.869982 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:49.971683 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:49.971634 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94d48cc-452b-418a-9d5f-64287cc7b716-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:49.971946 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:49.971764 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94d48cc-452b-418a-9d5f-64287cc7b716-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:49.971946 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:49.971802 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94d48cc-452b-418a-9d5f-64287cc7b716-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:49.971946 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:49.971845 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94d48cc-452b-418a-9d5f-64287cc7b716-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:49.972154 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:49.971965 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94d48cc-452b-418a-9d5f-64287cc7b716-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:49.972154 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:49.972082 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94d48cc-452b-418a-9d5f-64287cc7b716-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:49.972154 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:49.972132 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94d48cc-452b-418a-9d5f-64287cc7b716-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:49.972317 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:49.972217 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mltgg\" (UniqueName: \"kubernetes.io/projected/d94d48cc-452b-418a-9d5f-64287cc7b716-kube-api-access-mltgg\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:49.972317 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:49.972251 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94d48cc-452b-418a-9d5f-64287cc7b716-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:49.988611 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:49.988575 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6"] Nov 03 21:49:50.072922 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.072873 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94d48cc-452b-418a-9d5f-64287cc7b716-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.073103 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.072970 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94d48cc-452b-418a-9d5f-64287cc7b716-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.073103 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.073004 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94d48cc-452b-418a-9d5f-64287cc7b716-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.073103 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.073049 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94d48cc-452b-418a-9d5f-64287cc7b716-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.073103 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.073082 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94d48cc-452b-418a-9d5f-64287cc7b716-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.073346 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.073145 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94d48cc-452b-418a-9d5f-64287cc7b716-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.073346 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.073193 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94d48cc-452b-418a-9d5f-64287cc7b716-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.073346 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.073228 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mltgg\" (UniqueName: \"kubernetes.io/projected/d94d48cc-452b-418a-9d5f-64287cc7b716-kube-api-access-mltgg\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.073346 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.073284 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94d48cc-452b-418a-9d5f-64287cc7b716-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.073570 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.073445 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94d48cc-452b-418a-9d5f-64287cc7b716-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.073570 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.073476 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94d48cc-452b-418a-9d5f-64287cc7b716-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.073764 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.073740 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94d48cc-452b-418a-9d5f-64287cc7b716-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.073972 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.073781 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94d48cc-452b-418a-9d5f-64287cc7b716-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.073972 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.073884 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94d48cc-452b-418a-9d5f-64287cc7b716-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.075719 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.075697 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94d48cc-452b-418a-9d5f-64287cc7b716-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.075987 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.075968 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94d48cc-452b-418a-9d5f-64287cc7b716-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.136397 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.136321 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94d48cc-452b-418a-9d5f-64287cc7b716-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.136547 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.136394 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mltgg\" (UniqueName: \"kubernetes.io/projected/d94d48cc-452b-418a-9d5f-64287cc7b716-kube-api-access-mltgg\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jggj6\" (UID: \"d94d48cc-452b-418a-9d5f-64287cc7b716\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.185075 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.185041 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:50.246039 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.245996 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:50.438244 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.438135 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:50.619694 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.619666 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6"] Nov 03 21:49:50.620278 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:49:50.620251 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94d48cc_452b_418a_9d5f_64287cc7b716.slice/crio-bbf642e0cfefb4a5643f202c47639ee79bc2373515b05bb77f0e2ddf4aa2f2c9 WatchSource:0}: Error finding container bbf642e0cfefb4a5643f202c47639ee79bc2373515b05bb77f0e2ddf4aa2f2c9: Status 404 returned error can't find the container with id bbf642e0cfefb4a5643f202c47639ee79bc2373515b05bb77f0e2ddf4aa2f2c9 Nov 03 21:49:50.623062 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.623031 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:49:50.623148 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.623095 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:49:50.623148 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.623124 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:49:50.810802 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.810669 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" event={"ID":"d94d48cc-452b-418a-9d5f-64287cc7b716","Type":"ContainerStarted","Data":"b1934d96b56df0e596c82efd614f8ff2b34f5f040835f63bb567f6f2290c050c"} Nov 03 21:49:50.810802 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.810731 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" event={"ID":"d94d48cc-452b-418a-9d5f-64287cc7b716","Type":"ContainerStarted","Data":"bbf642e0cfefb4a5643f202c47639ee79bc2373515b05bb77f0e2ddf4aa2f2c9"} Nov 03 21:49:50.935967 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:50.935881 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" podStartSLOduration=1.9358673720000001 podStartE2EDuration="1.935867372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:50.93405021 +0000 UTC m=+3740.094122493" watchObservedRunningTime="2026-11-03 21:49:50.935867372 +0000 UTC m=+3740.095939644" Nov 03 21:49:51.072108 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.072017 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:51.185612 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.185572 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:49:51.187224 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.187197 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jggj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 21:49:51.187358 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.187245 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" podUID="d94d48cc-452b-418a-9d5f-64287cc7b716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 21:49:51.384498 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.384404 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd" podUID="c2e484f1-6067-45f8-a14e-e43fcfeab191" containerName="payload-pre-processing" containerID="cri-o://05b41ed187aba783f762d787ec666309da3582647548330470cd12a72e3390bc" gracePeriod=25 Nov 03 21:49:51.652579 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.652514 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd" Nov 03 21:49:51.696310 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.696271 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2e484f1-6067-45f8-a14e-e43fcfeab191-kube-api-access-krppq\" (UniqueName: \"kubernetes.io/projected/c2e484f1-6067-45f8-a14e-e43fcfeab191-kube-api-access-krppq\") pod \"c2e484f1-6067-45f8-a14e-e43fcfeab191\" (UID: \"c2e484f1-6067-45f8-a14e-e43fcfeab191\") " Nov 03 21:49:51.696501 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.696322 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2e484f1-6067-45f8-a14e-e43fcfeab191-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e484f1-6067-45f8-a14e-e43fcfeab191-plugins-config-volume\") pod \"c2e484f1-6067-45f8-a14e-e43fcfeab191\" (UID: \"c2e484f1-6067-45f8-a14e-e43fcfeab191\") " Nov 03 21:49:51.696729 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.696701 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e484f1-6067-45f8-a14e-e43fcfeab191-plugins-config-volume" pod "c2e484f1-6067-45f8-a14e-e43fcfeab191" (UID: "c2e484f1-6067-45f8-a14e-e43fcfeab191"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:49:51.698523 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.698500 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e484f1-6067-45f8-a14e-e43fcfeab191-kube-api-access-krppq" pod "c2e484f1-6067-45f8-a14e-e43fcfeab191" (UID: "c2e484f1-6067-45f8-a14e-e43fcfeab191"). InnerVolumeSpecName "kube-api-access-krppq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:51.798256 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.798104 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krppq\" (UniqueName: \"kubernetes.io/projected/c2e484f1-6067-45f8-a14e-e43fcfeab191-kube-api-access-krppq\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:51.798256 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.798147 4963 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e484f1-6067-45f8-a14e-e43fcfeab191-plugins-config-volume\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:49:51.818477 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.818445 4963 generic.go:356] "Generic (PLEG): container finished" podID="c2e484f1-6067-45f8-a14e-e43fcfeab191" containerID="05b41ed187aba783f762d787ec666309da3582647548330470cd12a72e3390bc" exitCode=0 Nov 03 21:49:51.818636 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.818491 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd" Nov 03 21:49:51.818636 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.818518 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd" event={"ID":"c2e484f1-6067-45f8-a14e-e43fcfeab191","Type":"ContainerDied","Data":"05b41ed187aba783f762d787ec666309da3582647548330470cd12a72e3390bc"} Nov 03 21:49:51.818636 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.818576 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd" event={"ID":"c2e484f1-6067-45f8-a14e-e43fcfeab191","Type":"ContainerDied","Data":"ac29a6af667960ea079a09b84c28a80e5258c8aabad8f8eac4f31f1219be89b9"} Nov 03 21:49:51.818636 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.818599 4963 scope.go:122] "RemoveContainer" containerID="05b41ed187aba783f762d787ec666309da3582647548330470cd12a72e3390bc" Nov 03 21:49:51.829736 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.829714 4963 scope.go:122] "RemoveContainer" containerID="05b41ed187aba783f762d787ec666309da3582647548330470cd12a72e3390bc" Nov 03 21:49:51.830095 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:49:51.830075 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b41ed187aba783f762d787ec666309da3582647548330470cd12a72e3390bc\": container with ID starting with 05b41ed187aba783f762d787ec666309da3582647548330470cd12a72e3390bc not found: ID does not exist" containerID="05b41ed187aba783f762d787ec666309da3582647548330470cd12a72e3390bc" Nov 03 21:49:51.830181 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.830105 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b41ed187aba783f762d787ec666309da3582647548330470cd12a72e3390bc"} err="failed to get container status \"05b41ed187aba783f762d787ec666309da3582647548330470cd12a72e3390bc\": rpc error: code = NotFound desc = could not find container \"05b41ed187aba783f762d787ec666309da3582647548330470cd12a72e3390bc\": container with ID starting with 05b41ed187aba783f762d787ec666309da3582647548330470cd12a72e3390bc not found: ID does not exist" Nov 03 21:49:51.940717 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.940638 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:51.999230 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:51.999186 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd"] Nov 03 21:49:52.006076 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:52.005411 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6a186d-7f68465587-4whkd"] Nov 03 21:49:52.190985 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:52.186293 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jggj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 21:49:52.190985 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:52.186356 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" podUID="d94d48cc-452b-418a-9d5f-64287cc7b716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 21:49:52.967572 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:52.967537 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2e484f1-6067-45f8-a14e-e43fcfeab191" path="/var/lib/kubelet/pods/c2e484f1-6067-45f8-a14e-e43fcfeab191/volumes" Nov 03 21:49:53.202929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:53.188331 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jggj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 21:49:53.202929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:53.188395 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" podUID="d94d48cc-452b-418a-9d5f-64287cc7b716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 21:49:54.193928 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:54.190292 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jggj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 21:49:54.193928 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:54.190354 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" podUID="d94d48cc-452b-418a-9d5f-64287cc7b716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 21:49:55.195929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:55.189319 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jggj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 21:49:55.195929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:55.189388 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" podUID="d94d48cc-452b-418a-9d5f-64287cc7b716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 21:49:55.975750 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:55.975714 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:49:56.200929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:56.190363 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jggj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 21:49:56.200929 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:56.190425 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" podUID="d94d48cc-452b-418a-9d5f-64287cc7b716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 21:49:57.200427 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:57.200386 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jggj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 21:49:57.200719 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:57.200689 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" podUID="d94d48cc-452b-418a-9d5f-64287cc7b716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 21:49:58.186373 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:58.186334 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jggj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 21:49:58.186941 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:49:58.186394 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" podUID="d94d48cc-452b-418a-9d5f-64287cc7b716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 21:50:00.168438 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:00.168400 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:50:00.168937 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:00.168831 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:50:00.170080 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:00.170062 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" Nov 03 21:50:00.718658 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:00.718620 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:05.391019 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:05.390973 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:05.585695 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:05.585644 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:06.076419 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:06.076380 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:06.349188 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:06.349103 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:10.036039 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:10.035999 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:10.224291 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:10.224255 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:10.422961 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:10.422821 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:12.134434 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:12.134399 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:12.529652 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:50:12.529612 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:50:13.407286 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:13.407252 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:13.600784 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:13.600748 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:14.098468 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.098436 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m"] Nov 03 21:50:14.099127 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.099106 4963 cpu_manager.go:414] "containerMap: removing container" podUID="c2e484f1-6067-45f8-a14e-e43fcfeab191" containerName="payload-pre-processing" Nov 03 21:50:14.099127 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.099127 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2e484f1-6067-45f8-a14e-e43fcfeab191" containerName="payload-pre-processing" Nov 03 21:50:14.099325 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.099267 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2e484f1-6067-45f8-a14e-e43fcfeab191" containerName="payload-pre-processing" Nov 03 21:50:14.103493 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.103468 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m" Nov 03 21:50:14.168357 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.168316 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m"] Nov 03 21:50:14.205353 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.205316 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-76eaf8-dockercfg-6zw59\"" Nov 03 21:50:14.205548 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.205388 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-76eaf8\"" Nov 03 21:50:14.216272 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.216237 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97nlp\" (UniqueName: \"kubernetes.io/projected/323445de-5ef5-4e36-8389-13c35281ce79-kube-api-access-97nlp\") pod \"payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m\" (UID: \"323445de-5ef5-4e36-8389-13c35281ce79\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m" Nov 03 21:50:14.216471 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.216410 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323445de-5ef5-4e36-8389-13c35281ce79-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m\" (UID: \"323445de-5ef5-4e36-8389-13c35281ce79\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m" Nov 03 21:50:14.317837 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.317785 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97nlp\" (UniqueName: \"kubernetes.io/projected/323445de-5ef5-4e36-8389-13c35281ce79-kube-api-access-97nlp\") pod \"payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m\" (UID: \"323445de-5ef5-4e36-8389-13c35281ce79\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m" Nov 03 21:50:14.318069 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.317971 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323445de-5ef5-4e36-8389-13c35281ce79-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m\" (UID: \"323445de-5ef5-4e36-8389-13c35281ce79\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m" Nov 03 21:50:14.318694 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.318671 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323445de-5ef5-4e36-8389-13c35281ce79-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m\" (UID: \"323445de-5ef5-4e36-8389-13c35281ce79\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m" Nov 03 21:50:14.381617 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.381531 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97nlp\" (UniqueName: \"kubernetes.io/projected/323445de-5ef5-4e36-8389-13c35281ce79-kube-api-access-97nlp\") pod \"payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m\" (UID: \"323445de-5ef5-4e36-8389-13c35281ce79\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m" Nov 03 21:50:14.417329 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.417295 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m" Nov 03 21:50:14.800788 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.800757 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m"] Nov 03 21:50:14.801377 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:50:14.801354 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323445de_5ef5_4e36_8389_13c35281ce79.slice/crio-11f6c2ec5980870dc2904a35c4aaf41bb9a1296da265ce4963a4a588b5bfdb9c WatchSource:0}: Error finding container 11f6c2ec5980870dc2904a35c4aaf41bb9a1296da265ce4963a4a588b5bfdb9c: Status 404 returned error can't find the container with id 11f6c2ec5980870dc2904a35c4aaf41bb9a1296da265ce4963a4a588b5bfdb9c Nov 03 21:50:14.995221 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.995183 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m" event={"ID":"323445de-5ef5-4e36-8389-13c35281ce79","Type":"ContainerStarted","Data":"8df5ec5813c8c17964894244ef54117b9f366d6587f33e8127a96a493fcf346b"} Nov 03 21:50:14.995395 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.995238 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m" event={"ID":"323445de-5ef5-4e36-8389-13c35281ce79","Type":"ContainerStarted","Data":"11f6c2ec5980870dc2904a35c4aaf41bb9a1296da265ce4963a4a588b5bfdb9c"} Nov 03 21:50:14.995395 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:14.995267 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m" Nov 03 21:50:15.024638 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:15.024603 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:15.116817 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:15.116702 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m" podStartSLOduration=1.116686385 podStartE2EDuration="1.116686385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:50:15.11503887 +0000 UTC m=+3764.275111153" watchObservedRunningTime="2026-11-03 21:50:15.116686385 +0000 UTC m=+3764.276758707" Nov 03 21:50:19.674982 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:19.674939 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6"] Nov 03 21:50:19.679186 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:19.679160 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:50:19.734318 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:19.734277 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6"] Nov 03 21:50:19.782391 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:19.782348 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a9a3e24-a4a2-4021-b951-1701bf172018-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6\" (UID: \"7a9a3e24-a4a2-4021-b951-1701bf172018\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:50:19.782391 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:19.782397 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9a3e24-a4a2-4021-b951-1701bf172018-metrics-certs\") pod \"maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6\" (UID: \"7a9a3e24-a4a2-4021-b951-1701bf172018\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:50:19.782604 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:19.782507 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a9a3e24-a4a2-4021-b951-1701bf172018-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6\" (UID: \"7a9a3e24-a4a2-4021-b951-1701bf172018\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:50:19.782604 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:19.782573 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brnhz\" (UniqueName: \"kubernetes.io/projected/7a9a3e24-a4a2-4021-b951-1701bf172018-kube-api-access-brnhz\") pod \"maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6\" (UID: \"7a9a3e24-a4a2-4021-b951-1701bf172018\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:50:19.782679 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:19.782649 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e7c432-serving-cert\"" Nov 03 21:50:19.883542 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:19.883503 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a9a3e24-a4a2-4021-b951-1701bf172018-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6\" (UID: \"7a9a3e24-a4a2-4021-b951-1701bf172018\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:50:19.883542 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:19.883549 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9a3e24-a4a2-4021-b951-1701bf172018-metrics-certs\") pod \"maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6\" (UID: \"7a9a3e24-a4a2-4021-b951-1701bf172018\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:50:19.883770 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:50:19.883653 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-serving-cert: secret "maas-api-e2e-inf-a-e7c432-serving-cert" not found Nov 03 21:50:19.883770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:19.883663 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a9a3e24-a4a2-4021-b951-1701bf172018-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6\" (UID: \"7a9a3e24-a4a2-4021-b951-1701bf172018\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:50:19.883770 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:50:19.883727 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9a3e24-a4a2-4021-b951-1701bf172018-maas-api-tls podName:7a9a3e24-a4a2-4021-b951-1701bf172018 nodeName:}" failed. No retries permitted until 2026-11-03 21:50:20.38370952 +0000 UTC m=+3769.543781770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a9a3e24-a4a2-4021-b951-1701bf172018-maas-api-tls") pod "maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" (UID: "7a9a3e24-a4a2-4021-b951-1701bf172018") : secret "maas-api-e2e-inf-a-e7c432-serving-cert" not found Nov 03 21:50:19.883770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:19.883766 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brnhz\" (UniqueName: \"kubernetes.io/projected/7a9a3e24-a4a2-4021-b951-1701bf172018-kube-api-access-brnhz\") pod \"maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6\" (UID: \"7a9a3e24-a4a2-4021-b951-1701bf172018\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:50:19.884378 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:19.884357 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a9a3e24-a4a2-4021-b951-1701bf172018-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6\" (UID: \"7a9a3e24-a4a2-4021-b951-1701bf172018\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:50:19.886148 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:19.886128 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9a3e24-a4a2-4021-b951-1701bf172018-metrics-certs\") pod \"maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6\" (UID: \"7a9a3e24-a4a2-4021-b951-1701bf172018\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:50:19.892237 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:19.892216 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:19.947672 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:19.947435 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brnhz\" (UniqueName: \"kubernetes.io/projected/7a9a3e24-a4a2-4021-b951-1701bf172018-kube-api-access-brnhz\") pod \"maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6\" (UID: \"7a9a3e24-a4a2-4021-b951-1701bf172018\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:50:20.084601 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:20.084567 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:20.273287 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:20.273207 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:20.391097 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:20.391044 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a9a3e24-a4a2-4021-b951-1701bf172018-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6\" (UID: \"7a9a3e24-a4a2-4021-b951-1701bf172018\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:50:20.393590 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:20.393570 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a9a3e24-a4a2-4021-b951-1701bf172018-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6\" (UID: \"7a9a3e24-a4a2-4021-b951-1701bf172018\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:50:20.593129 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:20.593043 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:50:20.973989 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:20.973956 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6"] Nov 03 21:50:20.974828 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:50:20.974804 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9a3e24_a4a2_4021_b951_1701bf172018.slice/crio-0abe30dae4ba21b54e98d6a635416479ebb37ca615c2dc54554956fe1851a5c4 WatchSource:0}: Error finding container 0abe30dae4ba21b54e98d6a635416479ebb37ca615c2dc54554956fe1851a5c4: Status 404 returned error can't find the container with id 0abe30dae4ba21b54e98d6a635416479ebb37ca615c2dc54554956fe1851a5c4 Nov 03 21:50:21.034256 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:21.034222 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" event={"ID":"7a9a3e24-a4a2-4021-b951-1701bf172018","Type":"ContainerStarted","Data":"0abe30dae4ba21b54e98d6a635416479ebb37ca615c2dc54554956fe1851a5c4"} Nov 03 21:50:22.042772 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:22.042731 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" event={"ID":"7a9a3e24-a4a2-4021-b951-1701bf172018","Type":"ContainerStarted","Data":"9bfbebb1bd44bc78305bd8cd3912cebe229279d243e564c3ed7da7e0d5d25b5c"} Nov 03 21:50:22.043217 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:22.042796 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:50:22.186283 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:22.186169 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" podStartSLOduration=2.238803052 podStartE2EDuration="3.186154234s" totalImagesPullingTime="947.351182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:50:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:50:22.184050209 +0000 UTC m=+3771.344122481" watchObservedRunningTime="2026-11-03 21:50:22.186154234 +0000 UTC m=+3771.346226505" Nov 03 21:50:24.587219 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:24.587188 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:26.005691 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:26.005661 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m" Nov 03 21:50:28.055385 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:28.055356 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:50:29.096417 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:29.096364 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:33.625224 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:33.625190 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:38.123003 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:38.122967 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:42.348327 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:42.348289 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:42.528410 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:50:42.528374 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:50:42.594612 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:42.594580 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:46.727807 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:46.727773 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc"] Nov 03 21:50:46.732098 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:46.732074 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:50:46.788394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:46.788352 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc"] Nov 03 21:50:46.794941 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:46.794868 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00dab90e-1071-496c-94cd-023bbd684cb6-maas-api-tls\") pod \"maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc\" (UID: \"00dab90e-1071-496c-94cd-023bbd684cb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:50:46.795146 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:46.794943 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00dab90e-1071-496c-94cd-023bbd684cb6-metrics-certs\") pod \"maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc\" (UID: \"00dab90e-1071-496c-94cd-023bbd684cb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:50:46.795146 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:46.795040 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00dab90e-1071-496c-94cd-023bbd684cb6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc\" (UID: \"00dab90e-1071-496c-94cd-023bbd684cb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:50:46.795146 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:46.795082 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmkst\" (UniqueName: \"kubernetes.io/projected/00dab90e-1071-496c-94cd-023bbd684cb6-kube-api-access-gmkst\") pod \"maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc\" (UID: \"00dab90e-1071-496c-94cd-023bbd684cb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:50:46.835586 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:46.835551 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-13c6b6-serving-cert\"" Nov 03 21:50:46.896203 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:46.896156 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00dab90e-1071-496c-94cd-023bbd684cb6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc\" (UID: \"00dab90e-1071-496c-94cd-023bbd684cb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:50:46.896390 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:46.896240 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmkst\" (UniqueName: \"kubernetes.io/projected/00dab90e-1071-496c-94cd-023bbd684cb6-kube-api-access-gmkst\") pod \"maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc\" (UID: \"00dab90e-1071-496c-94cd-023bbd684cb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:50:46.896439 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:46.896397 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00dab90e-1071-496c-94cd-023bbd684cb6-maas-api-tls\") pod \"maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc\" (UID: \"00dab90e-1071-496c-94cd-023bbd684cb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:50:46.896439 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:46.896431 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00dab90e-1071-496c-94cd-023bbd684cb6-metrics-certs\") pod \"maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc\" (UID: \"00dab90e-1071-496c-94cd-023bbd684cb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:50:46.897131 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:50:46.897097 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-serving-cert: secret "maas-api-e2e-inf-a-13c6b6-serving-cert" not found Nov 03 21:50:46.897348 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:50:46.897172 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00dab90e-1071-496c-94cd-023bbd684cb6-maas-api-tls podName:00dab90e-1071-496c-94cd-023bbd684cb6 nodeName:}" failed. No retries permitted until 2026-11-03 21:50:47.397150774 +0000 UTC m=+3796.557223039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00dab90e-1071-496c-94cd-023bbd684cb6-maas-api-tls") pod "maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" (UID: "00dab90e-1071-496c-94cd-023bbd684cb6") : secret "maas-api-e2e-inf-a-13c6b6-serving-cert" not found Nov 03 21:50:46.897645 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:46.897615 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00dab90e-1071-496c-94cd-023bbd684cb6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc\" (UID: \"00dab90e-1071-496c-94cd-023bbd684cb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:50:46.900657 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:46.900630 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00dab90e-1071-496c-94cd-023bbd684cb6-metrics-certs\") pod \"maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc\" (UID: \"00dab90e-1071-496c-94cd-023bbd684cb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:50:46.950808 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:46.950775 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:46.963685 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:46.963646 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmkst\" (UniqueName: \"kubernetes.io/projected/00dab90e-1071-496c-94cd-023bbd684cb6-kube-api-access-gmkst\") pod \"maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc\" (UID: \"00dab90e-1071-496c-94cd-023bbd684cb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:50:47.258894 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:47.258860 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:47.402492 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:47.402450 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00dab90e-1071-496c-94cd-023bbd684cb6-maas-api-tls\") pod \"maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc\" (UID: \"00dab90e-1071-496c-94cd-023bbd684cb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:50:47.405210 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:47.405174 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00dab90e-1071-496c-94cd-023bbd684cb6-maas-api-tls\") pod \"maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc\" (UID: \"00dab90e-1071-496c-94cd-023bbd684cb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:50:47.645788 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:47.645694 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:50:48.033415 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:48.033386 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc"] Nov 03 21:50:48.033833 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:50:48.033812 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00dab90e_1071_496c_94cd_023bbd684cb6.slice/crio-505d0eac4e3504b110728c5543940f9e825ea259bd4367886cbe3adcb1bace82 WatchSource:0}: Error finding container 505d0eac4e3504b110728c5543940f9e825ea259bd4367886cbe3adcb1bace82: Status 404 returned error can't find the container with id 505d0eac4e3504b110728c5543940f9e825ea259bd4367886cbe3adcb1bace82 Nov 03 21:50:48.203491 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:48.203455 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" event={"ID":"00dab90e-1071-496c-94cd-023bbd684cb6","Type":"ContainerStarted","Data":"505d0eac4e3504b110728c5543940f9e825ea259bd4367886cbe3adcb1bace82"} Nov 03 21:50:49.210975 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:49.210933 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" event={"ID":"00dab90e-1071-496c-94cd-023bbd684cb6","Type":"ContainerStarted","Data":"00d3ce055c675290dd503aacee81772f8c726ec1dfd18fca9eb4c53dfc0570ed"} Nov 03 21:50:49.211432 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:49.211006 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:50:49.335705 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:49.335654 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" podStartSLOduration=2.483535669 podStartE2EDuration="3.335639445s" totalImagesPullingTime="852.103776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:50:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:50:49.33222381 +0000 UTC m=+3798.492296083" watchObservedRunningTime="2026-11-03 21:50:49.335639445 +0000 UTC m=+3798.495711710" Nov 03 21:50:51.641818 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:51.641781 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:53.713085 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:53.713046 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:55.223048 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:55.223018 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:50:55.449716 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:55.449678 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh"] Nov 03 21:50:55.449987 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:55.449958 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" podUID="f2acbef8-8a3b-45f1-ac55-15212a819222" containerName="maas-api" containerID="cri-o://e3073fef26aa157acb8ac5ac8e4bbc31a695f53aff1420a17c2b12ae3bacb9b7" gracePeriod=30 Nov 03 21:50:55.581942 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:55.581884 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:50:55.741431 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:55.741403 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:50:55.903067 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:55.902985 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2acbef8-8a3b-45f1-ac55-15212a819222-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2acbef8-8a3b-45f1-ac55-15212a819222-openshift-service-ca\") pod \"f2acbef8-8a3b-45f1-ac55-15212a819222\" (UID: \"f2acbef8-8a3b-45f1-ac55-15212a819222\") " Nov 03 21:50:55.903067 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:55.903044 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2acbef8-8a3b-45f1-ac55-15212a819222-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2acbef8-8a3b-45f1-ac55-15212a819222-maas-api-tls\") pod \"f2acbef8-8a3b-45f1-ac55-15212a819222\" (UID: \"f2acbef8-8a3b-45f1-ac55-15212a819222\") " Nov 03 21:50:55.903310 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:55.903170 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2acbef8-8a3b-45f1-ac55-15212a819222-kube-api-access-fjqtf\" (UniqueName: \"kubernetes.io/projected/f2acbef8-8a3b-45f1-ac55-15212a819222-kube-api-access-fjqtf\") pod \"f2acbef8-8a3b-45f1-ac55-15212a819222\" (UID: \"f2acbef8-8a3b-45f1-ac55-15212a819222\") " Nov 03 21:50:55.903310 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:55.903268 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2acbef8-8a3b-45f1-ac55-15212a819222-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2acbef8-8a3b-45f1-ac55-15212a819222-metrics-certs\") pod \"f2acbef8-8a3b-45f1-ac55-15212a819222\" (UID: \"f2acbef8-8a3b-45f1-ac55-15212a819222\") " Nov 03 21:50:55.903417 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:55.903372 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2acbef8-8a3b-45f1-ac55-15212a819222-openshift-service-ca" pod "f2acbef8-8a3b-45f1-ac55-15212a819222" (UID: "f2acbef8-8a3b-45f1-ac55-15212a819222"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:50:55.903823 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:55.903797 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2acbef8-8a3b-45f1-ac55-15212a819222-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:50:55.905421 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:55.905391 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2acbef8-8a3b-45f1-ac55-15212a819222-kube-api-access-fjqtf" pod "f2acbef8-8a3b-45f1-ac55-15212a819222" (UID: "f2acbef8-8a3b-45f1-ac55-15212a819222"). InnerVolumeSpecName "kube-api-access-fjqtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:50:55.905770 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:55.905749 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2acbef8-8a3b-45f1-ac55-15212a819222-maas-api-tls" pod "f2acbef8-8a3b-45f1-ac55-15212a819222" (UID: "f2acbef8-8a3b-45f1-ac55-15212a819222"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:50:55.907072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:55.907053 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2acbef8-8a3b-45f1-ac55-15212a819222-metrics-certs" pod "f2acbef8-8a3b-45f1-ac55-15212a819222" (UID: "f2acbef8-8a3b-45f1-ac55-15212a819222"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:50:56.005522 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:56.005474 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2acbef8-8a3b-45f1-ac55-15212a819222-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:50:56.005522 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:56.005514 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjqtf\" (UniqueName: \"kubernetes.io/projected/f2acbef8-8a3b-45f1-ac55-15212a819222-kube-api-access-fjqtf\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:50:56.005522 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:56.005525 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2acbef8-8a3b-45f1-ac55-15212a819222-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:50:56.257508 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:56.257460 4963 generic.go:356] "Generic (PLEG): container finished" podID="f2acbef8-8a3b-45f1-ac55-15212a819222" containerID="e3073fef26aa157acb8ac5ac8e4bbc31a695f53aff1420a17c2b12ae3bacb9b7" exitCode=0 Nov 03 21:50:56.258026 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:56.257522 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" Nov 03 21:50:56.258026 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:56.257551 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" event={"ID":"f2acbef8-8a3b-45f1-ac55-15212a819222","Type":"ContainerDied","Data":"e3073fef26aa157acb8ac5ac8e4bbc31a695f53aff1420a17c2b12ae3bacb9b7"} Nov 03 21:50:56.258026 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:56.257628 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh" event={"ID":"f2acbef8-8a3b-45f1-ac55-15212a819222","Type":"ContainerDied","Data":"bf767f3e8ab8aa83be70b31f92af19af9011db590c64f3febe4e098d1062a113"} Nov 03 21:50:56.258026 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:56.257660 4963 scope.go:122] "RemoveContainer" containerID="e3073fef26aa157acb8ac5ac8e4bbc31a695f53aff1420a17c2b12ae3bacb9b7" Nov 03 21:50:56.269043 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:56.269017 4963 scope.go:122] "RemoveContainer" containerID="e3073fef26aa157acb8ac5ac8e4bbc31a695f53aff1420a17c2b12ae3bacb9b7" Nov 03 21:50:56.269331 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:50:56.269314 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3073fef26aa157acb8ac5ac8e4bbc31a695f53aff1420a17c2b12ae3bacb9b7\": container with ID starting with e3073fef26aa157acb8ac5ac8e4bbc31a695f53aff1420a17c2b12ae3bacb9b7 not found: ID does not exist" containerID="e3073fef26aa157acb8ac5ac8e4bbc31a695f53aff1420a17c2b12ae3bacb9b7" Nov 03 21:50:56.269379 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:56.269340 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3073fef26aa157acb8ac5ac8e4bbc31a695f53aff1420a17c2b12ae3bacb9b7"} err="failed to get container status \"e3073fef26aa157acb8ac5ac8e4bbc31a695f53aff1420a17c2b12ae3bacb9b7\": rpc error: code = NotFound desc = could not find container \"e3073fef26aa157acb8ac5ac8e4bbc31a695f53aff1420a17c2b12ae3bacb9b7\": container with ID starting with e3073fef26aa157acb8ac5ac8e4bbc31a695f53aff1420a17c2b12ae3bacb9b7 not found: ID does not exist" Nov 03 21:50:56.434113 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:56.434077 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh"] Nov 03 21:50:56.484130 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:56.484093 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a186d-5c98f568f9-bk8bh"] Nov 03 21:50:56.967707 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:56.967669 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2acbef8-8a3b-45f1-ac55-15212a819222" path="/var/lib/kubelet/pods/f2acbef8-8a3b-45f1-ac55-15212a819222/volumes" Nov 03 21:50:57.233885 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:50:57.233792 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:01.522213 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:01.522177 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:01.715711 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:01.715674 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:01.902712 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:01.902631 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:03.078398 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:03.078363 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:03.564654 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:03.564620 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:04.531948 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:04.531871 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:05.206803 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:05.206751 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:05.397769 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:05.397736 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:06.344068 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.344031 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd"] Nov 03 21:51:06.344602 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.344587 4963 cpu_manager.go:414] "containerMap: removing container" podUID="f2acbef8-8a3b-45f1-ac55-15212a819222" containerName="maas-api" Nov 03 21:51:06.344602 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.344602 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2acbef8-8a3b-45f1-ac55-15212a819222" containerName="maas-api" Nov 03 21:51:06.344712 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.344703 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2acbef8-8a3b-45f1-ac55-15212a819222" containerName="maas-api" Nov 03 21:51:06.348221 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.348181 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:51:06.405434 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.405386 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd"] Nov 03 21:51:06.439846 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.439812 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb25b496-0a84-4d07-9116-694a4a145511-maas-api-tls\") pod \"maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd\" (UID: \"eb25b496-0a84-4d07-9116-694a4a145511\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:51:06.440101 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.439867 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp6z7\" (UniqueName: \"kubernetes.io/projected/eb25b496-0a84-4d07-9116-694a4a145511-kube-api-access-wp6z7\") pod \"maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd\" (UID: \"eb25b496-0a84-4d07-9116-694a4a145511\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:51:06.440101 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.440025 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb25b496-0a84-4d07-9116-694a4a145511-openshift-service-ca\") pod \"maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd\" (UID: \"eb25b496-0a84-4d07-9116-694a4a145511\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:51:06.440227 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.440103 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb25b496-0a84-4d07-9116-694a4a145511-metrics-certs\") pod \"maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd\" (UID: \"eb25b496-0a84-4d07-9116-694a4a145511\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:51:06.452187 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.452155 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-965fe8-serving-cert\"" Nov 03 21:51:06.541008 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.540975 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb25b496-0a84-4d07-9116-694a4a145511-maas-api-tls\") pod \"maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd\" (UID: \"eb25b496-0a84-4d07-9116-694a4a145511\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:51:06.541295 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.541036 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp6z7\" (UniqueName: \"kubernetes.io/projected/eb25b496-0a84-4d07-9116-694a4a145511-kube-api-access-wp6z7\") pod \"maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd\" (UID: \"eb25b496-0a84-4d07-9116-694a4a145511\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:51:06.541295 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.541065 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb25b496-0a84-4d07-9116-694a4a145511-openshift-service-ca\") pod \"maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd\" (UID: \"eb25b496-0a84-4d07-9116-694a4a145511\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:51:06.541295 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:51:06.541125 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-serving-cert: secret "maas-api-e2e-inf-a-965fe8-serving-cert" not found Nov 03 21:51:06.541295 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:51:06.541201 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb25b496-0a84-4d07-9116-694a4a145511-maas-api-tls podName:eb25b496-0a84-4d07-9116-694a4a145511 nodeName:}" failed. No retries permitted until 2026-11-03 21:51:07.041183651 +0000 UTC m=+3816.201255919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb25b496-0a84-4d07-9116-694a4a145511-maas-api-tls") pod "maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" (UID: "eb25b496-0a84-4d07-9116-694a4a145511") : secret "maas-api-e2e-inf-a-965fe8-serving-cert" not found Nov 03 21:51:06.541295 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.541128 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb25b496-0a84-4d07-9116-694a4a145511-metrics-certs\") pod \"maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd\" (UID: \"eb25b496-0a84-4d07-9116-694a4a145511\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:51:06.541975 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.541952 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb25b496-0a84-4d07-9116-694a4a145511-openshift-service-ca\") pod \"maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd\" (UID: \"eb25b496-0a84-4d07-9116-694a4a145511\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:51:06.543665 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.543644 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb25b496-0a84-4d07-9116-694a4a145511-metrics-certs\") pod \"maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd\" (UID: \"eb25b496-0a84-4d07-9116-694a4a145511\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:51:06.555448 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.555422 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:06.604969 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.604876 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp6z7\" (UniqueName: \"kubernetes.io/projected/eb25b496-0a84-4d07-9116-694a4a145511-kube-api-access-wp6z7\") pod \"maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd\" (UID: \"eb25b496-0a84-4d07-9116-694a4a145511\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:51:06.744637 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.744603 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:06.926248 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:06.926167 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:07.048217 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:07.048182 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb25b496-0a84-4d07-9116-694a4a145511-maas-api-tls\") pod \"maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd\" (UID: \"eb25b496-0a84-4d07-9116-694a4a145511\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:51:07.050655 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:07.050626 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb25b496-0a84-4d07-9116-694a4a145511-maas-api-tls\") pod \"maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd\" (UID: \"eb25b496-0a84-4d07-9116-694a4a145511\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:51:07.265326 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:07.265292 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:51:07.648490 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:51:07.648459 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb25b496_0a84_4d07_9116_694a4a145511.slice/crio-c9418b13c8a303bd144fc7d75331afce16c5a71ab01f22f8f3c243ef97586f8e WatchSource:0}: Error finding container c9418b13c8a303bd144fc7d75331afce16c5a71ab01f22f8f3c243ef97586f8e: Status 404 returned error can't find the container with id c9418b13c8a303bd144fc7d75331afce16c5a71ab01f22f8f3c243ef97586f8e Nov 03 21:51:07.649209 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:07.649183 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd"] Nov 03 21:51:08.337695 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:08.337658 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" event={"ID":"eb25b496-0a84-4d07-9116-694a4a145511","Type":"ContainerStarted","Data":"c9418b13c8a303bd144fc7d75331afce16c5a71ab01f22f8f3c243ef97586f8e"} Nov 03 21:51:09.113496 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:09.113460 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:09.301000 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:09.300963 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:09.344889 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:09.344846 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" event={"ID":"eb25b496-0a84-4d07-9116-694a4a145511","Type":"ContainerStarted","Data":"2da7d13a282782d96b7fef2d16723dd1d98fb3963498c6e07b3fc82ccac68d81"} Nov 03 21:51:09.345081 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:09.344941 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:51:09.470235 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:09.470172 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" podStartSLOduration=2.588280534 podStartE2EDuration="3.470152805s" totalImagesPullingTime="881.872271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:51:09.469576629 +0000 UTC m=+3818.629648901" watchObservedRunningTime="2026-11-03 21:51:09.470152805 +0000 UTC m=+3818.630225077" Nov 03 21:51:11.155700 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:11.155610 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:11.611830 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:11.611795 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:12.359195 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:12.359157 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:12.531726 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:51:12.531690 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:51:12.603195 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:12.603160 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:15.356703 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:15.356673 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:51:16.463375 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:16.463338 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:21.312738 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:21.312701 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:23.038621 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:23.038587 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:26.105458 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:26.105402 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:30.964452 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:30.964411 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:35.580133 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:35.580094 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t"] Nov 03 21:51:35.583980 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:35.583958 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:51:35.639197 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:35.639163 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t"] Nov 03 21:51:35.687628 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:35.687595 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-76eaf8-serving-cert\"" Nov 03 21:51:35.690400 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:35.690366 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lgkl\" (UniqueName: \"kubernetes.io/projected/b37dbfc5-d401-4818-8e6e-038cac4013a9-kube-api-access-5lgkl\") pod \"maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t\" (UID: \"b37dbfc5-d401-4818-8e6e-038cac4013a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:51:35.690507 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:35.690477 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37dbfc5-d401-4818-8e6e-038cac4013a9-metrics-certs\") pod \"maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t\" (UID: \"b37dbfc5-d401-4818-8e6e-038cac4013a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:51:35.690573 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:35.690553 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b37dbfc5-d401-4818-8e6e-038cac4013a9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t\" (UID: \"b37dbfc5-d401-4818-8e6e-038cac4013a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:51:35.690630 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:35.690617 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b37dbfc5-d401-4818-8e6e-038cac4013a9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t\" (UID: \"b37dbfc5-d401-4818-8e6e-038cac4013a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:51:35.792309 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:35.792248 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lgkl\" (UniqueName: \"kubernetes.io/projected/b37dbfc5-d401-4818-8e6e-038cac4013a9-kube-api-access-5lgkl\") pod \"maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t\" (UID: \"b37dbfc5-d401-4818-8e6e-038cac4013a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:51:35.792541 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:35.792329 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37dbfc5-d401-4818-8e6e-038cac4013a9-metrics-certs\") pod \"maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t\" (UID: \"b37dbfc5-d401-4818-8e6e-038cac4013a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:51:35.792541 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:35.792354 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b37dbfc5-d401-4818-8e6e-038cac4013a9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t\" (UID: \"b37dbfc5-d401-4818-8e6e-038cac4013a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:51:35.792541 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:35.792383 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b37dbfc5-d401-4818-8e6e-038cac4013a9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t\" (UID: \"b37dbfc5-d401-4818-8e6e-038cac4013a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:51:35.792718 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:51:35.792545 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-serving-cert: secret "maas-api-e2e-extmod-b-76eaf8-serving-cert" not found Nov 03 21:51:35.792718 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:51:35.792609 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37dbfc5-d401-4818-8e6e-038cac4013a9-maas-api-tls podName:b37dbfc5-d401-4818-8e6e-038cac4013a9 nodeName:}" failed. No retries permitted until 2026-11-03 21:51:36.292591006 +0000 UTC m=+3845.452663274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b37dbfc5-d401-4818-8e6e-038cac4013a9-maas-api-tls") pod "maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" (UID: "b37dbfc5-d401-4818-8e6e-038cac4013a9") : secret "maas-api-e2e-extmod-b-76eaf8-serving-cert" not found Nov 03 21:51:35.793214 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:35.793186 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b37dbfc5-d401-4818-8e6e-038cac4013a9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t\" (UID: \"b37dbfc5-d401-4818-8e6e-038cac4013a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:51:35.795012 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:35.794994 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37dbfc5-d401-4818-8e6e-038cac4013a9-metrics-certs\") pod \"maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t\" (UID: \"b37dbfc5-d401-4818-8e6e-038cac4013a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:51:35.798674 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:35.798634 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:35.856552 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:35.856467 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lgkl\" (UniqueName: \"kubernetes.io/projected/b37dbfc5-d401-4818-8e6e-038cac4013a9-kube-api-access-5lgkl\") pod \"maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t\" (UID: \"b37dbfc5-d401-4818-8e6e-038cac4013a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:51:35.981847 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:35.981804 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:36.274774 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:36.274738 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:36.298403 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:36.298359 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b37dbfc5-d401-4818-8e6e-038cac4013a9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t\" (UID: \"b37dbfc5-d401-4818-8e6e-038cac4013a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:51:36.301162 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:36.301134 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b37dbfc5-d401-4818-8e6e-038cac4013a9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t\" (UID: \"b37dbfc5-d401-4818-8e6e-038cac4013a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:51:36.499024 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:36.498980 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:51:36.885150 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:36.885115 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t"] Nov 03 21:51:36.885598 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:51:36.885348 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37dbfc5_d401_4818_8e6e_038cac4013a9.slice/crio-498ad3dcb97be4003f66cf7d9a00dc630acce176242706812452c060dc777fd5 WatchSource:0}: Error finding container 498ad3dcb97be4003f66cf7d9a00dc630acce176242706812452c060dc777fd5: Status 404 returned error can't find the container with id 498ad3dcb97be4003f66cf7d9a00dc630acce176242706812452c060dc777fd5 Nov 03 21:51:37.536951 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:37.536913 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" event={"ID":"b37dbfc5-d401-4818-8e6e-038cac4013a9","Type":"ContainerStarted","Data":"498ad3dcb97be4003f66cf7d9a00dc630acce176242706812452c060dc777fd5"} Nov 03 21:51:38.544571 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:38.544534 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" event={"ID":"b37dbfc5-d401-4818-8e6e-038cac4013a9","Type":"ContainerStarted","Data":"aafb6abe9bb88466a0264dea4236cf7f43e3d47113d0205418bba617bde434b1"} Nov 03 21:51:38.545131 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:38.544600 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:51:38.668558 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:38.668499 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" podStartSLOduration=2.698009912 podStartE2EDuration="3.668483368s" totalImagesPullingTime="970.473456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:51:38.666264896 +0000 UTC m=+3847.826337168" watchObservedRunningTime="2026-11-03 21:51:38.668483368 +0000 UTC m=+3847.828555652" Nov 03 21:51:40.860227 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:40.860193 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:42.529027 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:51:42.528984 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:51:42.694298 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:42.694244 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:44.556832 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:44.556797 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:51:45.523679 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:45.523644 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:49.368319 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:49.368287 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:50.005917 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:50.005869 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:54.503759 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:54.503726 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:56.365884 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:56.365851 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:57.081348 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:57.081311 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:59.096673 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:59.096634 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:51:59.306968 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:51:59.306927 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:02.105808 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:02.105766 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:02.298368 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:02.298337 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:03.336948 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:03.336894 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:04.110868 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:04.110836 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:04.294466 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:04.294388 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:05.341266 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:05.341234 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:05.528580 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:05.528546 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:09.035727 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:09.035694 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:12.529066 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:52:12.529026 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:52:14.104651 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:14.104620 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:18.924639 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:18.924608 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:23.265255 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:23.265219 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:23.544191 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:23.544099 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:28.341914 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:28.341876 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:29.109893 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:29.109856 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:30.097657 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:30.097614 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:30.279416 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:30.279383 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:31.498918 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:31.498874 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:52:31.514421 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:31.514394 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:52:33.310321 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:33.310285 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:34.365956 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:34.365890 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:34.556212 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:34.556173 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:35.166699 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:35.166662 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:37.810863 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:37.810782 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:39.533469 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:39.533411 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:39.722765 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:39.722733 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:42.250338 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:42.250259 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:42.436570 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:42.436536 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:42.528659 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:52:42.528578 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:52:43.019588 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:43.019552 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:43.811854 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:43.811810 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:47.345839 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:47.345806 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:50.658434 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:50.658398 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:51.444727 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:51.444697 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:51.629876 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:51.629834 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:52.143500 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:52.143439 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:56.436688 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:56.436631 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:52:56.819667 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:52:56.819630 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:01.315666 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:01.315612 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:05.800449 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:05.800416 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:09.583185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:09.583145 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:10.401647 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:10.401609 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:12.529532 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:53:12.529497 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:53:14.858878 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:14.858836 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:15.554117 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:15.554081 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:17.076412 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:17.076376 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:17.267061 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:17.267029 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:19.661647 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:19.661616 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:21.189240 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:21.189201 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:24.311337 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:24.311300 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:28.818821 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:28.818777 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:33.324078 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:33.324041 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:37.539284 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:37.539248 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm"] Nov 03 21:53:37.543163 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:37.543142 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:53:37.600427 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:37.600386 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm"] Nov 03 21:53:37.647315 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:37.647283 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0a0da1-serving-cert\"" Nov 03 21:53:37.689436 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:37.689397 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ac5cc6d-e645-4255-8885-f9663b349513-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm\" (UID: \"2ac5cc6d-e645-4255-8885-f9663b349513\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:53:37.689644 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:37.689528 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac5cc6d-e645-4255-8885-f9663b349513-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm\" (UID: \"2ac5cc6d-e645-4255-8885-f9663b349513\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:53:37.689644 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:37.689631 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2mmr\" (UniqueName: \"kubernetes.io/projected/2ac5cc6d-e645-4255-8885-f9663b349513-kube-api-access-j2mmr\") pod \"maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm\" (UID: \"2ac5cc6d-e645-4255-8885-f9663b349513\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:53:37.689761 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:37.689657 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ac5cc6d-e645-4255-8885-f9663b349513-metrics-certs\") pod \"maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm\" (UID: \"2ac5cc6d-e645-4255-8885-f9663b349513\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:53:37.755307 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:37.755276 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:37.791385 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:37.791300 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac5cc6d-e645-4255-8885-f9663b349513-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm\" (UID: \"2ac5cc6d-e645-4255-8885-f9663b349513\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:53:37.791548 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:37.791381 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2mmr\" (UniqueName: \"kubernetes.io/projected/2ac5cc6d-e645-4255-8885-f9663b349513-kube-api-access-j2mmr\") pod \"maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm\" (UID: \"2ac5cc6d-e645-4255-8885-f9663b349513\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:53:37.791548 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:37.791411 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ac5cc6d-e645-4255-8885-f9663b349513-metrics-certs\") pod \"maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm\" (UID: \"2ac5cc6d-e645-4255-8885-f9663b349513\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:53:37.791644 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:37.791550 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ac5cc6d-e645-4255-8885-f9663b349513-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm\" (UID: \"2ac5cc6d-e645-4255-8885-f9663b349513\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:53:37.791724 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:53:37.791704 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-serving-cert: secret "maas-api-e2e-inf-b-0a0da1-serving-cert" not found Nov 03 21:53:37.791792 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:53:37.791782 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ac5cc6d-e645-4255-8885-f9663b349513-maas-api-tls podName:2ac5cc6d-e645-4255-8885-f9663b349513 nodeName:}" failed. No retries permitted until 2026-11-03 21:53:38.29176198 +0000 UTC m=+3967.451834236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ac5cc6d-e645-4255-8885-f9663b349513-maas-api-tls") pod "maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" (UID: "2ac5cc6d-e645-4255-8885-f9663b349513") : secret "maas-api-e2e-inf-b-0a0da1-serving-cert" not found Nov 03 21:53:37.792161 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:37.792138 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac5cc6d-e645-4255-8885-f9663b349513-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm\" (UID: \"2ac5cc6d-e645-4255-8885-f9663b349513\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:53:37.794069 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:37.794049 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ac5cc6d-e645-4255-8885-f9663b349513-metrics-certs\") pod \"maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm\" (UID: \"2ac5cc6d-e645-4255-8885-f9663b349513\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:53:37.855308 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:37.855268 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2mmr\" (UniqueName: \"kubernetes.io/projected/2ac5cc6d-e645-4255-8885-f9663b349513-kube-api-access-j2mmr\") pod \"maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm\" (UID: \"2ac5cc6d-e645-4255-8885-f9663b349513\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:53:37.937898 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:37.937868 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:38.120494 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:38.120397 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:38.297082 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:38.297042 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ac5cc6d-e645-4255-8885-f9663b349513-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm\" (UID: \"2ac5cc6d-e645-4255-8885-f9663b349513\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:53:38.299610 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:38.299589 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ac5cc6d-e645-4255-8885-f9663b349513-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm\" (UID: \"2ac5cc6d-e645-4255-8885-f9663b349513\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:53:38.456476 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:38.456443 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:53:38.841037 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:38.841011 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm"] Nov 03 21:53:38.841995 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:53:38.841958 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac5cc6d_e645_4255_8885_f9663b349513.slice/crio-fe02f32fa704b1f8f65b94614638e6a24ef8aec617da4cdb048c27ad38529406 WatchSource:0}: Error finding container fe02f32fa704b1f8f65b94614638e6a24ef8aec617da4cdb048c27ad38529406: Status 404 returned error can't find the container with id fe02f32fa704b1f8f65b94614638e6a24ef8aec617da4cdb048c27ad38529406 Nov 03 21:53:38.843992 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:38.843970 4963 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:53:39.335405 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:39.335369 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" event={"ID":"2ac5cc6d-e645-4255-8885-f9663b349513","Type":"ContainerStarted","Data":"fe02f32fa704b1f8f65b94614638e6a24ef8aec617da4cdb048c27ad38529406"} Nov 03 21:53:40.343384 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:40.343346 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" event={"ID":"2ac5cc6d-e645-4255-8885-f9663b349513","Type":"ContainerStarted","Data":"b527cb7a4924a19db05ac5bfdd8f7f5eb8a83e9c217c64fe9daa03974b8ee71d"} Nov 03 21:53:40.343799 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:40.343570 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:53:40.468824 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:40.468760 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" podStartSLOduration=2.561351659 podStartE2EDuration="3.468738167s" totalImagesPullingTime="907.386508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:53:40.465280501 +0000 UTC m=+3969.625352776" watchObservedRunningTime="2026-11-03 21:53:40.468738167 +0000 UTC m=+3969.628810440" Nov 03 21:53:42.452565 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:42.452531 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:42.531275 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:53:42.531230 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:53:43.348404 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:43.348372 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:46.357023 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:46.356993 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:53:46.996399 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:46.996362 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:51.506957 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:51.506920 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:53:56.134282 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:53:56.134245 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:00.618717 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:00.618678 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:05.305314 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:05.305275 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:08.687657 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:08.687625 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:09.138757 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.138718 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd"] Nov 03 21:54:09.139019 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.138993 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" podUID="eb25b496-0a84-4d07-9116-694a4a145511" containerName="maas-api" containerID="cri-o://2da7d13a282782d96b7fef2d16723dd1d98fb3963498c6e07b3fc82ccac68d81" gracePeriod=30 Nov 03 21:54:09.272685 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.272647 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:09.432897 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.432860 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:54:09.477893 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.477852 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb25b496-0a84-4d07-9116-694a4a145511-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb25b496-0a84-4d07-9116-694a4a145511-metrics-certs\") pod \"eb25b496-0a84-4d07-9116-694a4a145511\" (UID: \"eb25b496-0a84-4d07-9116-694a4a145511\") " Nov 03 21:54:09.478121 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.477928 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb25b496-0a84-4d07-9116-694a4a145511-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb25b496-0a84-4d07-9116-694a4a145511-openshift-service-ca\") pod \"eb25b496-0a84-4d07-9116-694a4a145511\" (UID: \"eb25b496-0a84-4d07-9116-694a4a145511\") " Nov 03 21:54:09.478527 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.478494 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb25b496-0a84-4d07-9116-694a4a145511-openshift-service-ca" pod "eb25b496-0a84-4d07-9116-694a4a145511" (UID: "eb25b496-0a84-4d07-9116-694a4a145511"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:54:09.480653 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.480625 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb25b496-0a84-4d07-9116-694a4a145511-metrics-certs" pod "eb25b496-0a84-4d07-9116-694a4a145511" (UID: "eb25b496-0a84-4d07-9116-694a4a145511"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:54:09.529323 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.529290 4963 generic.go:356] "Generic (PLEG): container finished" podID="eb25b496-0a84-4d07-9116-694a4a145511" containerID="2da7d13a282782d96b7fef2d16723dd1d98fb3963498c6e07b3fc82ccac68d81" exitCode=0 Nov 03 21:54:09.529543 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.529348 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" Nov 03 21:54:09.529543 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.529384 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" event={"ID":"eb25b496-0a84-4d07-9116-694a4a145511","Type":"ContainerDied","Data":"2da7d13a282782d96b7fef2d16723dd1d98fb3963498c6e07b3fc82ccac68d81"} Nov 03 21:54:09.529543 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.529436 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd" event={"ID":"eb25b496-0a84-4d07-9116-694a4a145511","Type":"ContainerDied","Data":"c9418b13c8a303bd144fc7d75331afce16c5a71ab01f22f8f3c243ef97586f8e"} Nov 03 21:54:09.529543 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.529466 4963 scope.go:122] "RemoveContainer" containerID="2da7d13a282782d96b7fef2d16723dd1d98fb3963498c6e07b3fc82ccac68d81" Nov 03 21:54:09.550344 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.550152 4963 scope.go:122] "RemoveContainer" containerID="2da7d13a282782d96b7fef2d16723dd1d98fb3963498c6e07b3fc82ccac68d81" Nov 03 21:54:09.550658 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:54:09.550631 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da7d13a282782d96b7fef2d16723dd1d98fb3963498c6e07b3fc82ccac68d81\": container with ID starting with 2da7d13a282782d96b7fef2d16723dd1d98fb3963498c6e07b3fc82ccac68d81 not found: ID does not exist" containerID="2da7d13a282782d96b7fef2d16723dd1d98fb3963498c6e07b3fc82ccac68d81" Nov 03 21:54:09.550739 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.550674 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da7d13a282782d96b7fef2d16723dd1d98fb3963498c6e07b3fc82ccac68d81"} err="failed to get container status \"2da7d13a282782d96b7fef2d16723dd1d98fb3963498c6e07b3fc82ccac68d81\": rpc error: code = NotFound desc = could not find container \"2da7d13a282782d96b7fef2d16723dd1d98fb3963498c6e07b3fc82ccac68d81\": container with ID starting with 2da7d13a282782d96b7fef2d16723dd1d98fb3963498c6e07b3fc82ccac68d81 not found: ID does not exist" Nov 03 21:54:09.579410 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.579373 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb25b496-0a84-4d07-9116-694a4a145511-kube-api-access-wp6z7\" (UniqueName: \"kubernetes.io/projected/eb25b496-0a84-4d07-9116-694a4a145511-kube-api-access-wp6z7\") pod \"eb25b496-0a84-4d07-9116-694a4a145511\" (UID: \"eb25b496-0a84-4d07-9116-694a4a145511\") " Nov 03 21:54:09.579605 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.579431 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb25b496-0a84-4d07-9116-694a4a145511-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb25b496-0a84-4d07-9116-694a4a145511-maas-api-tls\") pod \"eb25b496-0a84-4d07-9116-694a4a145511\" (UID: \"eb25b496-0a84-4d07-9116-694a4a145511\") " Nov 03 21:54:09.580005 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.579978 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb25b496-0a84-4d07-9116-694a4a145511-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:09.580130 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.580007 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb25b496-0a84-4d07-9116-694a4a145511-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:09.581669 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.581642 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb25b496-0a84-4d07-9116-694a4a145511-kube-api-access-wp6z7" pod "eb25b496-0a84-4d07-9116-694a4a145511" (UID: "eb25b496-0a84-4d07-9116-694a4a145511"). InnerVolumeSpecName "kube-api-access-wp6z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:54:09.581669 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.581653 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb25b496-0a84-4d07-9116-694a4a145511-maas-api-tls" pod "eb25b496-0a84-4d07-9116-694a4a145511" (UID: "eb25b496-0a84-4d07-9116-694a4a145511"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:54:09.680720 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.680626 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp6z7\" (UniqueName: \"kubernetes.io/projected/eb25b496-0a84-4d07-9116-694a4a145511-kube-api-access-wp6z7\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:09.680720 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.680666 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb25b496-0a84-4d07-9116-694a4a145511-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:09.956706 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.956672 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd"] Nov 03 21:54:09.960333 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:09.960310 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-965fe8-6c89589587-mmxwd"] Nov 03 21:54:10.928598 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:10.928558 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:10.969561 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:10.969528 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb25b496-0a84-4d07-9116-694a4a145511" path="/var/lib/kubelet/pods/eb25b496-0a84-4d07-9116-694a4a145511/volumes" Nov 03 21:54:12.500454 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:12.500414 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:12.532237 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:54:12.532194 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:54:12.681811 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:12.681773 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:14.422341 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:14.422302 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:14.956515 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:14.956483 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:15.136805 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:15.136767 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:15.752748 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:15.752716 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:15.937371 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:15.937340 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:16.120650 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:16.120565 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:17.558896 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:17.558856 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:17.789710 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:17.789671 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:20.655133 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:20.655100 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:20.837751 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:20.837714 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:24.766895 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:24.766859 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75"] Nov 03 21:54:24.767363 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:24.767085 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" podUID="15848a43-dcff-43db-801c-4ca6830d16d8" containerName="maas-api" containerID="cri-o://67a35bb03a26c9bb0d82912810640c99560f7193effffe602699fd2ac5227a23" gracePeriod=30 Nov 03 21:54:24.901161 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:24.901126 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:25.071051 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.071016 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:54:25.171356 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.171326 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15848a43-dcff-43db-801c-4ca6830d16d8-kube-api-access-6drd8\" (UniqueName: \"kubernetes.io/projected/15848a43-dcff-43db-801c-4ca6830d16d8-kube-api-access-6drd8\") pod \"15848a43-dcff-43db-801c-4ca6830d16d8\" (UID: \"15848a43-dcff-43db-801c-4ca6830d16d8\") " Nov 03 21:54:25.171572 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.171411 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15848a43-dcff-43db-801c-4ca6830d16d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15848a43-dcff-43db-801c-4ca6830d16d8-maas-api-tls\") pod \"15848a43-dcff-43db-801c-4ca6830d16d8\" (UID: \"15848a43-dcff-43db-801c-4ca6830d16d8\") " Nov 03 21:54:25.171666 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.171563 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15848a43-dcff-43db-801c-4ca6830d16d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15848a43-dcff-43db-801c-4ca6830d16d8-openshift-service-ca\") pod \"15848a43-dcff-43db-801c-4ca6830d16d8\" (UID: \"15848a43-dcff-43db-801c-4ca6830d16d8\") " Nov 03 21:54:25.171832 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.171812 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15848a43-dcff-43db-801c-4ca6830d16d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15848a43-dcff-43db-801c-4ca6830d16d8-metrics-certs\") pod \"15848a43-dcff-43db-801c-4ca6830d16d8\" (UID: \"15848a43-dcff-43db-801c-4ca6830d16d8\") " Nov 03 21:54:25.171972 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.171947 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15848a43-dcff-43db-801c-4ca6830d16d8-openshift-service-ca" pod "15848a43-dcff-43db-801c-4ca6830d16d8" (UID: "15848a43-dcff-43db-801c-4ca6830d16d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:54:25.172318 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.172294 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15848a43-dcff-43db-801c-4ca6830d16d8-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:25.174157 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.174129 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15848a43-dcff-43db-801c-4ca6830d16d8-maas-api-tls" pod "15848a43-dcff-43db-801c-4ca6830d16d8" (UID: "15848a43-dcff-43db-801c-4ca6830d16d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:54:25.174267 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.174183 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15848a43-dcff-43db-801c-4ca6830d16d8-metrics-certs" pod "15848a43-dcff-43db-801c-4ca6830d16d8" (UID: "15848a43-dcff-43db-801c-4ca6830d16d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:54:25.174267 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.174185 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15848a43-dcff-43db-801c-4ca6830d16d8-kube-api-access-6drd8" pod "15848a43-dcff-43db-801c-4ca6830d16d8" (UID: "15848a43-dcff-43db-801c-4ca6830d16d8"). InnerVolumeSpecName "kube-api-access-6drd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:54:25.274074 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.274030 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6drd8\" (UniqueName: \"kubernetes.io/projected/15848a43-dcff-43db-801c-4ca6830d16d8-kube-api-access-6drd8\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:25.274074 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.274068 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15848a43-dcff-43db-801c-4ca6830d16d8-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:25.274074 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.274081 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15848a43-dcff-43db-801c-4ca6830d16d8-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:25.632564 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.632529 4963 generic.go:356] "Generic (PLEG): container finished" podID="15848a43-dcff-43db-801c-4ca6830d16d8" containerID="67a35bb03a26c9bb0d82912810640c99560f7193effffe602699fd2ac5227a23" exitCode=0 Nov 03 21:54:25.632757 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.632579 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" Nov 03 21:54:25.632757 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.632611 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" event={"ID":"15848a43-dcff-43db-801c-4ca6830d16d8","Type":"ContainerDied","Data":"67a35bb03a26c9bb0d82912810640c99560f7193effffe602699fd2ac5227a23"} Nov 03 21:54:25.632757 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.632666 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75" event={"ID":"15848a43-dcff-43db-801c-4ca6830d16d8","Type":"ContainerDied","Data":"2575426bf6cac9184d674d87fafb8da4b326cf4ac6728049c142c45336e6688b"} Nov 03 21:54:25.632757 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.632688 4963 scope.go:122] "RemoveContainer" containerID="67a35bb03a26c9bb0d82912810640c99560f7193effffe602699fd2ac5227a23" Nov 03 21:54:25.644188 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.644164 4963 scope.go:122] "RemoveContainer" containerID="67a35bb03a26c9bb0d82912810640c99560f7193effffe602699fd2ac5227a23" Nov 03 21:54:25.644587 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:54:25.644567 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a35bb03a26c9bb0d82912810640c99560f7193effffe602699fd2ac5227a23\": container with ID starting with 67a35bb03a26c9bb0d82912810640c99560f7193effffe602699fd2ac5227a23 not found: ID does not exist" containerID="67a35bb03a26c9bb0d82912810640c99560f7193effffe602699fd2ac5227a23" Nov 03 21:54:25.644665 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.644607 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a35bb03a26c9bb0d82912810640c99560f7193effffe602699fd2ac5227a23"} err="failed to get container status \"67a35bb03a26c9bb0d82912810640c99560f7193effffe602699fd2ac5227a23\": rpc error: code = NotFound desc = could not find container \"67a35bb03a26c9bb0d82912810640c99560f7193effffe602699fd2ac5227a23\": container with ID starting with 67a35bb03a26c9bb0d82912810640c99560f7193effffe602699fd2ac5227a23 not found: ID does not exist" Nov 03 21:54:25.806107 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.806064 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75"] Nov 03 21:54:25.819029 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:25.818992 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-267053-5599d7f969-wzz75"] Nov 03 21:54:26.625254 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:26.625222 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:26.968684 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:26.968650 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15848a43-dcff-43db-801c-4ca6830d16d8" path="/var/lib/kubelet/pods/15848a43-dcff-43db-801c-4ca6830d16d8/volumes" Nov 03 21:54:28.845372 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:28.845333 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:29.041068 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:29.041025 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:30.356461 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:30.356423 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:30.852592 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:30.852483 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:31.674637 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:31.674597 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:31.861425 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:31.861386 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:36.589402 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:36.589344 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:38.080144 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:38.080110 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:38.398114 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:38.398034 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m"] Nov 03 21:54:38.705842 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:38.705811 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:38.894734 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:38.894701 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:41.331782 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:41.331754 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:42.535316 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:54:42.535276 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:54:43.398866 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:43.398812 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m" podUID="323445de-5ef5-4e36-8389-13c35281ce79" containerName="payload-pre-processing" containerID="cri-o://8df5ec5813c8c17964894244ef54117b9f366d6587f33e8127a96a493fcf346b" gracePeriod=25 Nov 03 21:54:43.689661 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:43.689626 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m" Nov 03 21:54:43.691795 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:43.691762 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:43.748358 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:43.748324 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/323445de-5ef5-4e36-8389-13c35281ce79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323445de-5ef5-4e36-8389-13c35281ce79-plugins-config-volume\") pod \"323445de-5ef5-4e36-8389-13c35281ce79\" (UID: \"323445de-5ef5-4e36-8389-13c35281ce79\") " Nov 03 21:54:43.748552 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:43.748489 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323445de-5ef5-4e36-8389-13c35281ce79-kube-api-access-97nlp\" (UniqueName: \"kubernetes.io/projected/323445de-5ef5-4e36-8389-13c35281ce79-kube-api-access-97nlp\") pod \"323445de-5ef5-4e36-8389-13c35281ce79\" (UID: \"323445de-5ef5-4e36-8389-13c35281ce79\") " Nov 03 21:54:43.748745 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:43.748723 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323445de-5ef5-4e36-8389-13c35281ce79-plugins-config-volume" pod "323445de-5ef5-4e36-8389-13c35281ce79" (UID: "323445de-5ef5-4e36-8389-13c35281ce79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:54:43.748884 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:43.748864 4963 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323445de-5ef5-4e36-8389-13c35281ce79-plugins-config-volume\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:43.750679 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:43.750659 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323445de-5ef5-4e36-8389-13c35281ce79-kube-api-access-97nlp" pod "323445de-5ef5-4e36-8389-13c35281ce79" (UID: "323445de-5ef5-4e36-8389-13c35281ce79"). InnerVolumeSpecName "kube-api-access-97nlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:54:43.786373 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:43.786338 4963 generic.go:356] "Generic (PLEG): container finished" podID="323445de-5ef5-4e36-8389-13c35281ce79" containerID="8df5ec5813c8c17964894244ef54117b9f366d6587f33e8127a96a493fcf346b" exitCode=0 Nov 03 21:54:43.786556 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:43.786385 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m" Nov 03 21:54:43.786556 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:43.786406 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m" event={"ID":"323445de-5ef5-4e36-8389-13c35281ce79","Type":"ContainerDied","Data":"8df5ec5813c8c17964894244ef54117b9f366d6587f33e8127a96a493fcf346b"} Nov 03 21:54:43.786556 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:43.786467 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m" event={"ID":"323445de-5ef5-4e36-8389-13c35281ce79","Type":"ContainerDied","Data":"11f6c2ec5980870dc2904a35c4aaf41bb9a1296da265ce4963a4a588b5bfdb9c"} Nov 03 21:54:43.786556 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:43.786498 4963 scope.go:122] "RemoveContainer" containerID="8df5ec5813c8c17964894244ef54117b9f366d6587f33e8127a96a493fcf346b" Nov 03 21:54:43.797466 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:43.797437 4963 scope.go:122] "RemoveContainer" containerID="8df5ec5813c8c17964894244ef54117b9f366d6587f33e8127a96a493fcf346b" Nov 03 21:54:43.797791 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:54:43.797767 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df5ec5813c8c17964894244ef54117b9f366d6587f33e8127a96a493fcf346b\": container with ID starting with 8df5ec5813c8c17964894244ef54117b9f366d6587f33e8127a96a493fcf346b not found: ID does not exist" containerID="8df5ec5813c8c17964894244ef54117b9f366d6587f33e8127a96a493fcf346b" Nov 03 21:54:43.797841 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:43.797800 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df5ec5813c8c17964894244ef54117b9f366d6587f33e8127a96a493fcf346b"} err="failed to get container status \"8df5ec5813c8c17964894244ef54117b9f366d6587f33e8127a96a493fcf346b\": rpc error: code = NotFound desc = could not find container \"8df5ec5813c8c17964894244ef54117b9f366d6587f33e8127a96a493fcf346b\": container with ID starting with 8df5ec5813c8c17964894244ef54117b9f366d6587f33e8127a96a493fcf346b not found: ID does not exist" Nov 03 21:54:43.850369 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:43.850334 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97nlp\" (UniqueName: \"kubernetes.io/projected/323445de-5ef5-4e36-8389-13c35281ce79-kube-api-access-97nlp\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:54:43.928700 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:43.928611 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m"] Nov 03 21:54:43.932900 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:43.932869 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-76eaf8-79c757ddc-7pz2m"] Nov 03 21:54:44.305514 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:44.305461 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:44.968772 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:44.968738 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="323445de-5ef5-4e36-8389-13c35281ce79" path="/var/lib/kubelet/pods/323445de-5ef5-4e36-8389-13c35281ce79/volumes" Nov 03 21:54:45.905170 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:45.905122 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:50.388205 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:50.388166 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:54.977373 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:54.977336 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:54:59.472604 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:54:59.472561 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:01.446920 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:01.446870 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:04.010900 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:04.010866 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:07.102867 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:07.102818 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:07.291848 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:07.291808 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:08.064104 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:08.064063 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:08.729548 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:08.729515 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:12.528837 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:55:12.528792 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:55:13.600144 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:13.600114 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:13.780005 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:13.779973 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:18.462117 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:18.462081 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:22.566702 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:22.566669 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc"] Nov 03 21:55:22.567489 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:22.566873 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" podUID="00dab90e-1071-496c-94cd-023bbd684cb6" containerName="maas-api" containerID="cri-o://00d3ce055c675290dd503aacee81772f8c726ec1dfd18fca9eb4c53dfc0570ed" gracePeriod=30 Nov 03 21:55:22.707716 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:22.707679 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:22.882667 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:22.882637 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:55:22.921260 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:22.921220 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00dab90e-1071-496c-94cd-023bbd684cb6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00dab90e-1071-496c-94cd-023bbd684cb6-metrics-certs\") pod \"00dab90e-1071-496c-94cd-023bbd684cb6\" (UID: \"00dab90e-1071-496c-94cd-023bbd684cb6\") " Nov 03 21:55:22.921464 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:22.921377 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00dab90e-1071-496c-94cd-023bbd684cb6-kube-api-access-gmkst\" (UniqueName: \"kubernetes.io/projected/00dab90e-1071-496c-94cd-023bbd684cb6-kube-api-access-gmkst\") pod \"00dab90e-1071-496c-94cd-023bbd684cb6\" (UID: \"00dab90e-1071-496c-94cd-023bbd684cb6\") " Nov 03 21:55:22.921464 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:22.921419 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00dab90e-1071-496c-94cd-023bbd684cb6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00dab90e-1071-496c-94cd-023bbd684cb6-openshift-service-ca\") pod \"00dab90e-1071-496c-94cd-023bbd684cb6\" (UID: \"00dab90e-1071-496c-94cd-023bbd684cb6\") " Nov 03 21:55:22.921601 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:22.921466 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00dab90e-1071-496c-94cd-023bbd684cb6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00dab90e-1071-496c-94cd-023bbd684cb6-maas-api-tls\") pod \"00dab90e-1071-496c-94cd-023bbd684cb6\" (UID: \"00dab90e-1071-496c-94cd-023bbd684cb6\") " Nov 03 21:55:22.922016 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:22.921977 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00dab90e-1071-496c-94cd-023bbd684cb6-openshift-service-ca" pod "00dab90e-1071-496c-94cd-023bbd684cb6" (UID: "00dab90e-1071-496c-94cd-023bbd684cb6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:55:22.924086 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:22.924058 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00dab90e-1071-496c-94cd-023bbd684cb6-maas-api-tls" pod "00dab90e-1071-496c-94cd-023bbd684cb6" (UID: "00dab90e-1071-496c-94cd-023bbd684cb6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:55:22.924186 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:22.924088 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00dab90e-1071-496c-94cd-023bbd684cb6-metrics-certs" pod "00dab90e-1071-496c-94cd-023bbd684cb6" (UID: "00dab90e-1071-496c-94cd-023bbd684cb6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:55:22.924186 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:22.924111 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00dab90e-1071-496c-94cd-023bbd684cb6-kube-api-access-gmkst" pod "00dab90e-1071-496c-94cd-023bbd684cb6" (UID: "00dab90e-1071-496c-94cd-023bbd684cb6"). InnerVolumeSpecName "kube-api-access-gmkst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:55:23.023625 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:23.023499 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmkst\" (UniqueName: \"kubernetes.io/projected/00dab90e-1071-496c-94cd-023bbd684cb6-kube-api-access-gmkst\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:55:23.023834 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:23.023808 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00dab90e-1071-496c-94cd-023bbd684cb6-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:55:23.023834 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:23.023833 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00dab90e-1071-496c-94cd-023bbd684cb6-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:55:23.024029 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:23.023845 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00dab90e-1071-496c-94cd-023bbd684cb6-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:55:23.033569 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:23.033538 4963 generic.go:356] "Generic (PLEG): container finished" podID="00dab90e-1071-496c-94cd-023bbd684cb6" containerID="00d3ce055c675290dd503aacee81772f8c726ec1dfd18fca9eb4c53dfc0570ed" exitCode=0 Nov 03 21:55:23.033813 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:23.033590 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" Nov 03 21:55:23.033813 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:23.033615 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" event={"ID":"00dab90e-1071-496c-94cd-023bbd684cb6","Type":"ContainerDied","Data":"00d3ce055c675290dd503aacee81772f8c726ec1dfd18fca9eb4c53dfc0570ed"} Nov 03 21:55:23.033813 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:23.033677 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc" event={"ID":"00dab90e-1071-496c-94cd-023bbd684cb6","Type":"ContainerDied","Data":"505d0eac4e3504b110728c5543940f9e825ea259bd4367886cbe3adcb1bace82"} Nov 03 21:55:23.033813 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:23.033706 4963 scope.go:122] "RemoveContainer" containerID="00d3ce055c675290dd503aacee81772f8c726ec1dfd18fca9eb4c53dfc0570ed" Nov 03 21:55:23.046017 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:23.045996 4963 scope.go:122] "RemoveContainer" containerID="00d3ce055c675290dd503aacee81772f8c726ec1dfd18fca9eb4c53dfc0570ed" Nov 03 21:55:23.046344 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:55:23.046320 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d3ce055c675290dd503aacee81772f8c726ec1dfd18fca9eb4c53dfc0570ed\": container with ID starting with 00d3ce055c675290dd503aacee81772f8c726ec1dfd18fca9eb4c53dfc0570ed not found: ID does not exist" containerID="00d3ce055c675290dd503aacee81772f8c726ec1dfd18fca9eb4c53dfc0570ed" Nov 03 21:55:23.046441 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:23.046353 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d3ce055c675290dd503aacee81772f8c726ec1dfd18fca9eb4c53dfc0570ed"} err="failed to get container status \"00d3ce055c675290dd503aacee81772f8c726ec1dfd18fca9eb4c53dfc0570ed\": rpc error: code = NotFound desc = could not find container \"00d3ce055c675290dd503aacee81772f8c726ec1dfd18fca9eb4c53dfc0570ed\": container with ID starting with 00d3ce055c675290dd503aacee81772f8c726ec1dfd18fca9eb4c53dfc0570ed not found: ID does not exist" Nov 03 21:55:23.206826 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:23.206788 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc"] Nov 03 21:55:23.211306 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:23.211278 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-13c6b6-8598fbcc75-llpcc"] Nov 03 21:55:24.407307 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:24.407277 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:24.589440 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:24.589409 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:24.896957 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:24.896924 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:24.974166 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:24.974126 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00dab90e-1071-496c-94cd-023bbd684cb6" path="/var/lib/kubelet/pods/00dab90e-1071-496c-94cd-023bbd684cb6/volumes" Nov 03 21:55:27.270069 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:27.270039 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:27.465706 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:27.465676 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:29.269312 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:29.269281 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:29.451074 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:29.451045 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:32.384153 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:32.384120 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:34.370089 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:34.370057 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:38.328620 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:38.328582 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:38.854796 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:38.854760 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:40.045137 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:40.045063 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:40.238257 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:40.238223 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:40.746640 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:40.746601 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:42.528813 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:55:42.528773 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:55:43.303070 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:43.303036 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:43.487012 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:43.486978 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:43.702159 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:43.702124 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:44.047720 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:44.047639 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:44.614615 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:44.614584 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:44.897414 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:44.897330 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:45.091601 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:45.091570 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:47.708846 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:47.708813 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:47.894132 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:47.894099 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:49.624150 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:49.624115 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:49.922195 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:49.922078 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:54.116126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:54.116087 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:57.566483 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:57.566449 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:57.891752 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:57.891668 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t"] Nov 03 21:55:57.891962 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:57.891869 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" podUID="b37dbfc5-d401-4818-8e6e-038cac4013a9" containerName="maas-api" containerID="cri-o://aafb6abe9bb88466a0264dea4236cf7f43e3d47113d0205418bba617bde434b1" gracePeriod=30 Nov 03 21:55:58.026722 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.026688 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:55:58.179581 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.179548 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:55:58.248517 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.248477 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37dbfc5-d401-4818-8e6e-038cac4013a9-kube-api-access-5lgkl\" (UniqueName: \"kubernetes.io/projected/b37dbfc5-d401-4818-8e6e-038cac4013a9-kube-api-access-5lgkl\") pod \"b37dbfc5-d401-4818-8e6e-038cac4013a9\" (UID: \"b37dbfc5-d401-4818-8e6e-038cac4013a9\") " Nov 03 21:55:58.248729 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.248566 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37dbfc5-d401-4818-8e6e-038cac4013a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b37dbfc5-d401-4818-8e6e-038cac4013a9-openshift-service-ca\") pod \"b37dbfc5-d401-4818-8e6e-038cac4013a9\" (UID: \"b37dbfc5-d401-4818-8e6e-038cac4013a9\") " Nov 03 21:55:58.248729 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.248692 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b37dbfc5-d401-4818-8e6e-038cac4013a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37dbfc5-d401-4818-8e6e-038cac4013a9-metrics-certs\") pod \"b37dbfc5-d401-4818-8e6e-038cac4013a9\" (UID: \"b37dbfc5-d401-4818-8e6e-038cac4013a9\") " Nov 03 21:55:58.248860 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.248772 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b37dbfc5-d401-4818-8e6e-038cac4013a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b37dbfc5-d401-4818-8e6e-038cac4013a9-maas-api-tls\") pod \"b37dbfc5-d401-4818-8e6e-038cac4013a9\" (UID: \"b37dbfc5-d401-4818-8e6e-038cac4013a9\") " Nov 03 21:55:58.248956 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.248935 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37dbfc5-d401-4818-8e6e-038cac4013a9-openshift-service-ca" pod "b37dbfc5-d401-4818-8e6e-038cac4013a9" (UID: "b37dbfc5-d401-4818-8e6e-038cac4013a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:55:58.249317 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.249296 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b37dbfc5-d401-4818-8e6e-038cac4013a9-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:55:58.251260 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.251223 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b37dbfc5-d401-4818-8e6e-038cac4013a9-maas-api-tls" pod "b37dbfc5-d401-4818-8e6e-038cac4013a9" (UID: "b37dbfc5-d401-4818-8e6e-038cac4013a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:55:58.251507 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.251478 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b37dbfc5-d401-4818-8e6e-038cac4013a9-metrics-certs" pod "b37dbfc5-d401-4818-8e6e-038cac4013a9" (UID: "b37dbfc5-d401-4818-8e6e-038cac4013a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:55:58.251653 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.251631 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37dbfc5-d401-4818-8e6e-038cac4013a9-kube-api-access-5lgkl" pod "b37dbfc5-d401-4818-8e6e-038cac4013a9" (UID: "b37dbfc5-d401-4818-8e6e-038cac4013a9"). InnerVolumeSpecName "kube-api-access-5lgkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:55:58.253553 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.253524 4963 generic.go:356] "Generic (PLEG): container finished" podID="b37dbfc5-d401-4818-8e6e-038cac4013a9" containerID="aafb6abe9bb88466a0264dea4236cf7f43e3d47113d0205418bba617bde434b1" exitCode=0 Nov 03 21:55:58.253645 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.253589 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" Nov 03 21:55:58.253645 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.253611 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" event={"ID":"b37dbfc5-d401-4818-8e6e-038cac4013a9","Type":"ContainerDied","Data":"aafb6abe9bb88466a0264dea4236cf7f43e3d47113d0205418bba617bde434b1"} Nov 03 21:55:58.253737 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.253675 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t" event={"ID":"b37dbfc5-d401-4818-8e6e-038cac4013a9","Type":"ContainerDied","Data":"498ad3dcb97be4003f66cf7d9a00dc630acce176242706812452c060dc777fd5"} Nov 03 21:55:58.253737 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.253707 4963 scope.go:122] "RemoveContainer" containerID="aafb6abe9bb88466a0264dea4236cf7f43e3d47113d0205418bba617bde434b1" Nov 03 21:55:58.289877 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.289851 4963 scope.go:122] "RemoveContainer" containerID="aafb6abe9bb88466a0264dea4236cf7f43e3d47113d0205418bba617bde434b1" Nov 03 21:55:58.290382 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:55:58.290354 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aafb6abe9bb88466a0264dea4236cf7f43e3d47113d0205418bba617bde434b1\": container with ID starting with aafb6abe9bb88466a0264dea4236cf7f43e3d47113d0205418bba617bde434b1 not found: ID does not exist" containerID="aafb6abe9bb88466a0264dea4236cf7f43e3d47113d0205418bba617bde434b1" Nov 03 21:55:58.290513 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.290389 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aafb6abe9bb88466a0264dea4236cf7f43e3d47113d0205418bba617bde434b1"} err="failed to get container status \"aafb6abe9bb88466a0264dea4236cf7f43e3d47113d0205418bba617bde434b1\": rpc error: code = NotFound desc = could not find container \"aafb6abe9bb88466a0264dea4236cf7f43e3d47113d0205418bba617bde434b1\": container with ID starting with aafb6abe9bb88466a0264dea4236cf7f43e3d47113d0205418bba617bde434b1 not found: ID does not exist" Nov 03 21:55:58.350391 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.350354 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37dbfc5-d401-4818-8e6e-038cac4013a9-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:55:58.350391 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.350386 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b37dbfc5-d401-4818-8e6e-038cac4013a9-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:55:58.350391 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.350396 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lgkl\" (UniqueName: \"kubernetes.io/projected/b37dbfc5-d401-4818-8e6e-038cac4013a9-kube-api-access-5lgkl\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:55:58.419058 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.418939 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t"] Nov 03 21:55:58.423606 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.423567 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-76eaf8-d869ffbd-97r5t"] Nov 03 21:55:58.967634 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:58.967595 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37dbfc5-d401-4818-8e6e-038cac4013a9" path="/var/lib/kubelet/pods/b37dbfc5-d401-4818-8e6e-038cac4013a9/volumes" Nov 03 21:55:59.706013 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:55:59.705982 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:00.291442 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:00.291403 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:01.788371 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:01.788337 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:01.967247 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:01.967210 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:03.186193 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:03.186156 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:03.711273 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:03.711242 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:04.108707 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:04.108628 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:04.351646 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:04.351607 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:04.782096 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:04.782061 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:04.962169 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:04.962094 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:05.332958 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:05.332892 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:08.404624 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:08.404588 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6"] Nov 03 21:56:08.405122 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:08.404834 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" podUID="7a9a3e24-a4a2-4021-b951-1701bf172018" containerName="maas-api" containerID="cri-o://9bfbebb1bd44bc78305bd8cd3912cebe229279d243e564c3ed7da7e0d5d25b5c" gracePeriod=30 Nov 03 21:56:08.537854 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:08.537816 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:08.690717 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:08.690689 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:56:08.770249 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:08.770213 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a9a3e24-a4a2-4021-b951-1701bf172018-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9a3e24-a4a2-4021-b951-1701bf172018-metrics-certs\") pod \"7a9a3e24-a4a2-4021-b951-1701bf172018\" (UID: \"7a9a3e24-a4a2-4021-b951-1701bf172018\") " Nov 03 21:56:08.770467 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:08.770324 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a9a3e24-a4a2-4021-b951-1701bf172018-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a9a3e24-a4a2-4021-b951-1701bf172018-maas-api-tls\") pod \"7a9a3e24-a4a2-4021-b951-1701bf172018\" (UID: \"7a9a3e24-a4a2-4021-b951-1701bf172018\") " Nov 03 21:56:08.770467 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:08.770368 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a9a3e24-a4a2-4021-b951-1701bf172018-kube-api-access-brnhz\" (UniqueName: \"kubernetes.io/projected/7a9a3e24-a4a2-4021-b951-1701bf172018-kube-api-access-brnhz\") pod \"7a9a3e24-a4a2-4021-b951-1701bf172018\" (UID: \"7a9a3e24-a4a2-4021-b951-1701bf172018\") " Nov 03 21:56:08.770467 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:08.770459 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a9a3e24-a4a2-4021-b951-1701bf172018-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a9a3e24-a4a2-4021-b951-1701bf172018-openshift-service-ca\") pod \"7a9a3e24-a4a2-4021-b951-1701bf172018\" (UID: \"7a9a3e24-a4a2-4021-b951-1701bf172018\") " Nov 03 21:56:08.771639 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:08.771580 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a9a3e24-a4a2-4021-b951-1701bf172018-openshift-service-ca" pod "7a9a3e24-a4a2-4021-b951-1701bf172018" (UID: "7a9a3e24-a4a2-4021-b951-1701bf172018"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:56:08.774126 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:08.774080 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a9a3e24-a4a2-4021-b951-1701bf172018-maas-api-tls" pod "7a9a3e24-a4a2-4021-b951-1701bf172018" (UID: "7a9a3e24-a4a2-4021-b951-1701bf172018"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:56:08.781505 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:08.781469 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a9a3e24-a4a2-4021-b951-1701bf172018-metrics-certs" pod "7a9a3e24-a4a2-4021-b951-1701bf172018" (UID: "7a9a3e24-a4a2-4021-b951-1701bf172018"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:56:08.782087 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:08.782059 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9a3e24-a4a2-4021-b951-1701bf172018-kube-api-access-brnhz" pod "7a9a3e24-a4a2-4021-b951-1701bf172018" (UID: "7a9a3e24-a4a2-4021-b951-1701bf172018"). InnerVolumeSpecName "kube-api-access-brnhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:56:08.871793 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:08.871754 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brnhz\" (UniqueName: \"kubernetes.io/projected/7a9a3e24-a4a2-4021-b951-1701bf172018-kube-api-access-brnhz\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:56:08.871793 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:08.871786 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a9a3e24-a4a2-4021-b951-1701bf172018-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:56:08.871793 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:08.871797 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9a3e24-a4a2-4021-b951-1701bf172018-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:56:08.872109 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:08.871806 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a9a3e24-a4a2-4021-b951-1701bf172018-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:56:09.333317 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:09.333220 4963 generic.go:356] "Generic (PLEG): container finished" podID="7a9a3e24-a4a2-4021-b951-1701bf172018" containerID="9bfbebb1bd44bc78305bd8cd3912cebe229279d243e564c3ed7da7e0d5d25b5c" exitCode=0 Nov 03 21:56:09.333317 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:09.333302 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" Nov 03 21:56:09.333632 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:09.333306 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" event={"ID":"7a9a3e24-a4a2-4021-b951-1701bf172018","Type":"ContainerDied","Data":"9bfbebb1bd44bc78305bd8cd3912cebe229279d243e564c3ed7da7e0d5d25b5c"} Nov 03 21:56:09.333632 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:09.333365 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6" event={"ID":"7a9a3e24-a4a2-4021-b951-1701bf172018","Type":"ContainerDied","Data":"0abe30dae4ba21b54e98d6a635416479ebb37ca615c2dc54554956fe1851a5c4"} Nov 03 21:56:09.333632 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:09.333396 4963 scope.go:122] "RemoveContainer" containerID="9bfbebb1bd44bc78305bd8cd3912cebe229279d243e564c3ed7da7e0d5d25b5c" Nov 03 21:56:09.343849 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:09.343829 4963 scope.go:122] "RemoveContainer" containerID="9bfbebb1bd44bc78305bd8cd3912cebe229279d243e564c3ed7da7e0d5d25b5c" Nov 03 21:56:09.344159 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:56:09.344135 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bfbebb1bd44bc78305bd8cd3912cebe229279d243e564c3ed7da7e0d5d25b5c\": container with ID starting with 9bfbebb1bd44bc78305bd8cd3912cebe229279d243e564c3ed7da7e0d5d25b5c not found: ID does not exist" containerID="9bfbebb1bd44bc78305bd8cd3912cebe229279d243e564c3ed7da7e0d5d25b5c" Nov 03 21:56:09.344233 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:09.344166 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bfbebb1bd44bc78305bd8cd3912cebe229279d243e564c3ed7da7e0d5d25b5c"} err="failed to get container status \"9bfbebb1bd44bc78305bd8cd3912cebe229279d243e564c3ed7da7e0d5d25b5c\": rpc error: code = NotFound desc = could not find container \"9bfbebb1bd44bc78305bd8cd3912cebe229279d243e564c3ed7da7e0d5d25b5c\": container with ID starting with 9bfbebb1bd44bc78305bd8cd3912cebe229279d243e564c3ed7da7e0d5d25b5c not found: ID does not exist" Nov 03 21:56:09.504584 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:09.504547 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6"] Nov 03 21:56:09.508302 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:09.508276 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c432-5d97ff97cb-2tvf6"] Nov 03 21:56:10.168403 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:10.168363 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:10.968353 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:10.968316 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a9a3e24-a4a2-4021-b951-1701bf172018" path="/var/lib/kubelet/pods/7a9a3e24-a4a2-4021-b951-1701bf172018/volumes" Nov 03 21:56:11.833457 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:11.833423 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:12.023201 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:12.023154 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:12.478256 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:12.478218 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:12.533941 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:56:12.533881 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:56:14.577611 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:14.577573 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:14.764083 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:14.764041 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:14.948405 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:14.948378 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:16.183998 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:16.183967 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:16.691532 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:16.691501 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:17.484382 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:17.484347 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:18.497595 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:18.497557 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:18.684006 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:18.683967 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:19.692241 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:19.692194 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:23.801137 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:23.801074 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:24.360990 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:24.360960 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:24.542600 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:24.542567 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:29.082197 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:29.082140 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:29.450914 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:29.450879 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:33.532121 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:33.532078 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:37.958862 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:37.958828 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:42.425250 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:42.425205 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:42.534086 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:56:42.534044 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:56:44.389887 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:44.389845 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:47.000285 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:47.000252 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:51.469734 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:51.469694 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:56:56.004740 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:56:56.004701 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:00.520446 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:00.520409 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:02.838931 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:02.838868 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:03.038827 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:03.038799 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:03.230351 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:03.230315 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:03.801630 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:03.801590 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:05.020689 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:05.020655 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:09.609700 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:09.609671 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:12.527570 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:57:12.527486 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:57:13.855266 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:13.855215 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:14.042860 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:14.042831 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:14.230370 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:14.230330 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:18.606577 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:18.606539 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:22.185025 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:22.184986 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:23.145206 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:23.145173 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:24.746763 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:24.746730 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:26.990484 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:26.990382 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm"] Nov 03 21:57:26.993506 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:26.990589 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" podUID="2ac5cc6d-e645-4255-8885-f9663b349513" containerName="maas-api" containerID="cri-o://b527cb7a4924a19db05ac5bfdd8f7f5eb8a83e9c217c64fe9daa03974b8ee71d" gracePeriod=30 Nov 03 21:57:27.069160 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.069124 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:27.275352 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.275315 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:57:27.418843 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.418804 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac5cc6d-e645-4255-8885-f9663b349513-kube-api-access-j2mmr\" (UniqueName: \"kubernetes.io/projected/2ac5cc6d-e645-4255-8885-f9663b349513-kube-api-access-j2mmr\") pod \"2ac5cc6d-e645-4255-8885-f9663b349513\" (UID: \"2ac5cc6d-e645-4255-8885-f9663b349513\") " Nov 03 21:57:27.419058 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.418876 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ac5cc6d-e645-4255-8885-f9663b349513-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ac5cc6d-e645-4255-8885-f9663b349513-maas-api-tls\") pod \"2ac5cc6d-e645-4255-8885-f9663b349513\" (UID: \"2ac5cc6d-e645-4255-8885-f9663b349513\") " Nov 03 21:57:27.419058 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.418961 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ac5cc6d-e645-4255-8885-f9663b349513-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac5cc6d-e645-4255-8885-f9663b349513-openshift-service-ca\") pod \"2ac5cc6d-e645-4255-8885-f9663b349513\" (UID: \"2ac5cc6d-e645-4255-8885-f9663b349513\") " Nov 03 21:57:27.419058 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.418998 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ac5cc6d-e645-4255-8885-f9663b349513-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ac5cc6d-e645-4255-8885-f9663b349513-metrics-certs\") pod \"2ac5cc6d-e645-4255-8885-f9663b349513\" (UID: \"2ac5cc6d-e645-4255-8885-f9663b349513\") " Nov 03 21:57:27.419410 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.419380 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac5cc6d-e645-4255-8885-f9663b349513-openshift-service-ca" pod "2ac5cc6d-e645-4255-8885-f9663b349513" (UID: "2ac5cc6d-e645-4255-8885-f9663b349513"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:57:27.422036 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.421159 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac5cc6d-e645-4255-8885-f9663b349513-metrics-certs" pod "2ac5cc6d-e645-4255-8885-f9663b349513" (UID: "2ac5cc6d-e645-4255-8885-f9663b349513"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:57:27.422036 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.421214 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac5cc6d-e645-4255-8885-f9663b349513-kube-api-access-j2mmr" pod "2ac5cc6d-e645-4255-8885-f9663b349513" (UID: "2ac5cc6d-e645-4255-8885-f9663b349513"). InnerVolumeSpecName "kube-api-access-j2mmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:57:27.422036 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.421242 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac5cc6d-e645-4255-8885-f9663b349513-maas-api-tls" pod "2ac5cc6d-e645-4255-8885-f9663b349513" (UID: "2ac5cc6d-e645-4255-8885-f9663b349513"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:57:27.521039 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.520956 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ac5cc6d-e645-4255-8885-f9663b349513-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:57:27.521039 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.520988 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac5cc6d-e645-4255-8885-f9663b349513-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:57:27.521039 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.520999 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ac5cc6d-e645-4255-8885-f9663b349513-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:57:27.521039 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.521010 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2mmr\" (UniqueName: \"kubernetes.io/projected/2ac5cc6d-e645-4255-8885-f9663b349513-kube-api-access-j2mmr\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:57:27.863214 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.863122 4963 generic.go:356] "Generic (PLEG): container finished" podID="2ac5cc6d-e645-4255-8885-f9663b349513" containerID="b527cb7a4924a19db05ac5bfdd8f7f5eb8a83e9c217c64fe9daa03974b8ee71d" exitCode=0 Nov 03 21:57:27.863214 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.863180 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" Nov 03 21:57:27.863214 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.863178 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" event={"ID":"2ac5cc6d-e645-4255-8885-f9663b349513","Type":"ContainerDied","Data":"b527cb7a4924a19db05ac5bfdd8f7f5eb8a83e9c217c64fe9daa03974b8ee71d"} Nov 03 21:57:27.863495 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.863264 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm" event={"ID":"2ac5cc6d-e645-4255-8885-f9663b349513","Type":"ContainerDied","Data":"fe02f32fa704b1f8f65b94614638e6a24ef8aec617da4cdb048c27ad38529406"} Nov 03 21:57:27.863495 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.863288 4963 scope.go:122] "RemoveContainer" containerID="b527cb7a4924a19db05ac5bfdd8f7f5eb8a83e9c217c64fe9daa03974b8ee71d" Nov 03 21:57:27.874558 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.874539 4963 scope.go:122] "RemoveContainer" containerID="b527cb7a4924a19db05ac5bfdd8f7f5eb8a83e9c217c64fe9daa03974b8ee71d" Nov 03 21:57:27.874848 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:57:27.874824 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b527cb7a4924a19db05ac5bfdd8f7f5eb8a83e9c217c64fe9daa03974b8ee71d\": container with ID starting with b527cb7a4924a19db05ac5bfdd8f7f5eb8a83e9c217c64fe9daa03974b8ee71d not found: ID does not exist" containerID="b527cb7a4924a19db05ac5bfdd8f7f5eb8a83e9c217c64fe9daa03974b8ee71d" Nov 03 21:57:27.874940 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:27.874855 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b527cb7a4924a19db05ac5bfdd8f7f5eb8a83e9c217c64fe9daa03974b8ee71d"} err="failed to get container status \"b527cb7a4924a19db05ac5bfdd8f7f5eb8a83e9c217c64fe9daa03974b8ee71d\": rpc error: code = NotFound desc = could not find container \"b527cb7a4924a19db05ac5bfdd8f7f5eb8a83e9c217c64fe9daa03974b8ee71d\": container with ID starting with b527cb7a4924a19db05ac5bfdd8f7f5eb8a83e9c217c64fe9daa03974b8ee71d not found: ID does not exist" Nov 03 21:57:28.035092 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:28.035057 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm"] Nov 03 21:57:28.039220 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:28.039190 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a0da1-684c98946d-jhsgm"] Nov 03 21:57:28.704190 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:28.704127 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:28.968299 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:28.968205 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ac5cc6d-e645-4255-8885-f9663b349513" path="/var/lib/kubelet/pods/2ac5cc6d-e645-4255-8885-f9663b349513/volumes" Nov 03 21:57:29.164181 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:29.164141 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:29.350275 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:29.350181 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:30.079278 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:30.079240 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:30.672177 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:30.672139 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:31.166564 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:31.166515 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:31.541750 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:31.541715 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:57:31.561695 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:31.561666 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 21:57:33.480669 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:33.480632 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:33.919355 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:33.919275 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:35.764111 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:35.764043 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:36.485497 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:36.485464 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:37.947283 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:37.947206 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:39.311948 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:39.311895 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:42.412122 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:42.412086 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:42.529937 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:57:42.529887 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:57:42.598274 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:42.598245 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:44.739090 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:44.739052 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:45.625185 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:45.625152 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:46.883239 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:46.883205 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:51.388882 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:51.388849 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:57:55.826314 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:57:55.826265 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:00.330764 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:58:00.330730 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:04.820136 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:58:04.820073 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:09.376753 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:58:09.376716 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:12.531747 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:58:12.531689 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:58:13.992495 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:58:13.992456 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:18.661473 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:58:18.661432 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:23.058290 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:58:23.058249 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:27.553762 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:58:27.553701 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:32.026754 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:58:32.026719 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:36.639521 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:58:36.639428 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:41.095835 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:58:41.095749 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:42.528935 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:58:42.528864 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:58:43.776564 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:58:43.776504 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:45.092296 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:58:45.092254 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:45.558601 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:58:45.558563 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:48.388934 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:58:48.388870 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:48.568941 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:58:48.568894 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:50.048202 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:58:50.048165 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:54.554871 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:58:54.554837 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:58:59.031704 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:58:59.031664 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:03.627141 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:03.627099 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:08.211169 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:08.211124 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:10.031491 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:10.031456 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:11.511137 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:11.511102 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:11.778025 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:11.777941 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:12.528296 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:59:12.528254 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:59:12.702321 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:12.702289 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:16.177312 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:16.177277 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:16.357103 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:16.357071 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:17.402678 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:17.402642 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:18.217952 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:18.217898 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:19.423306 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:19.423274 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:19.606006 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:19.605962 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:22.079658 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:22.079622 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:26.609631 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:26.609600 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:31.187751 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:31.187714 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:35.633386 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:35.633353 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:39.024469 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:39.024423 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:39.209366 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:39.209325 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:40.032761 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:40.032729 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:42.529495 ip-10-0-130-134 kubenswrapper[4963]: E1103 21:59:42.529458 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 21:59:44.498196 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:44.498163 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:44.682629 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:44.682595 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:45.436447 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:45.436417 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:49.028213 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.028170 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:49.611643 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.611607 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw"] Nov 03 21:59:49.612151 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612136 4963 cpu_manager.go:414] "containerMap: removing container" podUID="eb25b496-0a84-4d07-9116-694a4a145511" containerName="maas-api" Nov 03 21:59:49.612203 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612151 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb25b496-0a84-4d07-9116-694a4a145511" containerName="maas-api" Nov 03 21:59:49.612203 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612169 4963 cpu_manager.go:414] "containerMap: removing container" podUID="2ac5cc6d-e645-4255-8885-f9663b349513" containerName="maas-api" Nov 03 21:59:49.612203 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612175 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ac5cc6d-e645-4255-8885-f9663b349513" containerName="maas-api" Nov 03 21:59:49.612203 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612189 4963 cpu_manager.go:414] "containerMap: removing container" podUID="00dab90e-1071-496c-94cd-023bbd684cb6" containerName="maas-api" Nov 03 21:59:49.612203 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612194 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00dab90e-1071-496c-94cd-023bbd684cb6" containerName="maas-api" Nov 03 21:59:49.612203 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612204 4963 cpu_manager.go:414] "containerMap: removing container" podUID="323445de-5ef5-4e36-8389-13c35281ce79" containerName="payload-pre-processing" Nov 03 21:59:49.612394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612209 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323445de-5ef5-4e36-8389-13c35281ce79" containerName="payload-pre-processing" Nov 03 21:59:49.612394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612217 4963 cpu_manager.go:414] "containerMap: removing container" podUID="b37dbfc5-d401-4818-8e6e-038cac4013a9" containerName="maas-api" Nov 03 21:59:49.612394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612222 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37dbfc5-d401-4818-8e6e-038cac4013a9" containerName="maas-api" Nov 03 21:59:49.612394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612238 4963 cpu_manager.go:414] "containerMap: removing container" podUID="15848a43-dcff-43db-801c-4ca6830d16d8" containerName="maas-api" Nov 03 21:59:49.612394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612242 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15848a43-dcff-43db-801c-4ca6830d16d8" containerName="maas-api" Nov 03 21:59:49.612394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612248 4963 cpu_manager.go:414] "containerMap: removing container" podUID="7a9a3e24-a4a2-4021-b951-1701bf172018" containerName="maas-api" Nov 03 21:59:49.612394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612254 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a9a3e24-a4a2-4021-b951-1701bf172018" containerName="maas-api" Nov 03 21:59:49.612394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612328 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="323445de-5ef5-4e36-8389-13c35281ce79" containerName="payload-pre-processing" Nov 03 21:59:49.612394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612339 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37dbfc5-d401-4818-8e6e-038cac4013a9" containerName="maas-api" Nov 03 21:59:49.612394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612347 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="15848a43-dcff-43db-801c-4ca6830d16d8" containerName="maas-api" Nov 03 21:59:49.612394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612353 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a9a3e24-a4a2-4021-b951-1701bf172018" containerName="maas-api" Nov 03 21:59:49.612394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612363 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb25b496-0a84-4d07-9116-694a4a145511" containerName="maas-api" Nov 03 21:59:49.612394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612370 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ac5cc6d-e645-4255-8885-f9663b349513" containerName="maas-api" Nov 03 21:59:49.612394 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.612376 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="00dab90e-1071-496c-94cd-023bbd684cb6" containerName="maas-api" Nov 03 21:59:49.615775 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.615759 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.670375 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.670334 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw"] Nov 03 21:59:49.712269 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.712229 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-tmp-dir\") pod \"rate-test-model-9818d8-kserve-69b8cc59d5-xldvw\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.712449 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.712274 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-home\") pod \"rate-test-model-9818d8-kserve-69b8cc59d5-xldvw\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.712449 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.712418 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsjk4\" (UniqueName: \"kubernetes.io/projected/7eef8d7a-0034-40d0-9027-d6c4528dd24e-kube-api-access-vsjk4\") pod \"rate-test-model-9818d8-kserve-69b8cc59d5-xldvw\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.712564 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.712550 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eef8d7a-0034-40d0-9027-d6c4528dd24e-tls-certs\") pod \"rate-test-model-9818d8-kserve-69b8cc59d5-xldvw\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.712601 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.712591 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-dshm\") pod \"rate-test-model-9818d8-kserve-69b8cc59d5-xldvw\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.712681 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.712667 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-model-cache\") pod \"rate-test-model-9818d8-kserve-69b8cc59d5-xldvw\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.719592 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.719565 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-9818d8\"/\"kube-root-ca.crt\"" Nov 03 21:59:49.719745 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.719619 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-9818d8\"/\"openshift-service-ca.crt\"" Nov 03 21:59:49.722574 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.722557 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-9818d8\"/\"default-dockercfg-74qdw\"" Nov 03 21:59:49.722845 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.722827 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-9818d8\"/\"rate-test-model-9818d8-kserve-self-signed-certs\"" Nov 03 21:59:49.741179 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.741148 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:49.814021 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.813985 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-dshm\") pod \"rate-test-model-9818d8-kserve-69b8cc59d5-xldvw\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.814229 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.814045 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-model-cache\") pod \"rate-test-model-9818d8-kserve-69b8cc59d5-xldvw\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.814229 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.814093 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-tmp-dir\") pod \"rate-test-model-9818d8-kserve-69b8cc59d5-xldvw\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.814229 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.814119 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-home\") pod \"rate-test-model-9818d8-kserve-69b8cc59d5-xldvw\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.814229 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.814177 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsjk4\" (UniqueName: \"kubernetes.io/projected/7eef8d7a-0034-40d0-9027-d6c4528dd24e-kube-api-access-vsjk4\") pod \"rate-test-model-9818d8-kserve-69b8cc59d5-xldvw\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.814429 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.814308 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eef8d7a-0034-40d0-9027-d6c4528dd24e-tls-certs\") pod \"rate-test-model-9818d8-kserve-69b8cc59d5-xldvw\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.814560 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.814540 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-model-cache\") pod \"rate-test-model-9818d8-kserve-69b8cc59d5-xldvw\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.814664 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.814639 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-tmp-dir\") pod \"rate-test-model-9818d8-kserve-69b8cc59d5-xldvw\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.814738 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.814669 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-home\") pod \"rate-test-model-9818d8-kserve-69b8cc59d5-xldvw\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.816414 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.816389 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-dshm\") pod \"rate-test-model-9818d8-kserve-69b8cc59d5-xldvw\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.816892 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.816873 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eef8d7a-0034-40d0-9027-d6c4528dd24e-tls-certs\") pod \"rate-test-model-9818d8-kserve-69b8cc59d5-xldvw\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.877756 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.877691 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsjk4\" (UniqueName: \"kubernetes.io/projected/7eef8d7a-0034-40d0-9027-d6c4528dd24e-kube-api-access-vsjk4\") pod \"rate-test-model-9818d8-kserve-69b8cc59d5-xldvw\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:49.929158 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:49.929121 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:50.322779 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:50.322736 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw"] Nov 03 21:59:50.323222 ip-10-0-130-134 kubenswrapper[4963]: W1103 21:59:50.322824 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7eef8d7a_0034_40d0_9027_d6c4528dd24e.slice/crio-5d09ac227d3ed290dfeec9b46c3ea42e9260b0e3a042655374df5fa14edc3ea0 WatchSource:0}: Error finding container 5d09ac227d3ed290dfeec9b46c3ea42e9260b0e3a042655374df5fa14edc3ea0: Status 404 returned error can't find the container with id 5d09ac227d3ed290dfeec9b46c3ea42e9260b0e3a042655374df5fa14edc3ea0 Nov 03 21:59:50.787932 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:50.787882 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" event={"ID":"7eef8d7a-0034-40d0-9027-d6c4528dd24e","Type":"ContainerStarted","Data":"bbca12f72451053ee1650ee2c7804197fa2f0988ae39a577274c02736b4370cb"} Nov 03 21:59:50.788150 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:50.787941 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" event={"ID":"7eef8d7a-0034-40d0-9027-d6c4528dd24e","Type":"ContainerStarted","Data":"5d09ac227d3ed290dfeec9b46c3ea42e9260b0e3a042655374df5fa14edc3ea0"} Nov 03 21:59:50.912205 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:50.912125 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" podStartSLOduration=1.912106804 podStartE2EDuration="1.912106804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:50.908692569 +0000 UTC m=+4340.068764841" watchObservedRunningTime="2026-11-03 21:59:50.912106804 +0000 UTC m=+4340.072179079" Nov 03 21:59:50.929781 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:50.929744 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:51.174072 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:51.173987 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:53.642081 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:53.642048 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:58.112191 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:58.112138 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 21:59:59.929552 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:59.929503 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 21:59:59.942501 ip-10-0-130-134 kubenswrapper[4963]: I1103 21:59:59.942470 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 22:00:00.228426 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.228381 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn"] Nov 03 22:00:00.234351 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.234307 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2"] Nov 03 22:00:00.234534 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.234445 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" Nov 03 22:00:00.238653 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.238627 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll"] Nov 03 22:00:00.238800 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.238732 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" Nov 03 22:00:00.242963 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.242933 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" Nov 03 22:00:00.244448 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.244424 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v"] Nov 03 22:00:00.248076 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.248057 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm"] Nov 03 22:00:00.248195 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.248176 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" Nov 03 22:00:00.251866 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.251843 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" Nov 03 22:00:00.340693 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.340646 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn"] Nov 03 22:00:00.344805 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.344717 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2"] Nov 03 22:00:00.345430 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.345402 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7a8dfdc-4732-4c52-914a-53f510144184-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2\" (UID: \"f7a8dfdc-4732-4c52-914a-53f510144184\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" Nov 03 22:00:00.345543 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.345509 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5dpj\" (UniqueName: \"kubernetes.io/projected/f7a8dfdc-4732-4c52-914a-53f510144184-kube-api-access-x5dpj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2\" (UID: \"f7a8dfdc-4732-4c52-914a-53f510144184\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" Nov 03 22:00:00.345615 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.345569 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzvm7\" (UniqueName: \"kubernetes.io/projected/a9f51ebd-e158-4364-95c2-bedc152fc4b4-kube-api-access-wzvm7\") pod \"maas-api-key-cleanup-29895720-tkwzm\" (UID: \"a9f51ebd-e158-4364-95c2-bedc152fc4b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" Nov 03 22:00:00.345615 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.345597 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7c5507-a94f-4427-b503-248a20c75484-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v\" (UID: \"2d7c5507-a94f-4427-b503-248a20c75484\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" Nov 03 22:00:00.345733 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.345632 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f51ebd-e158-4364-95c2-bedc152fc4b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895720-tkwzm\" (UID: \"a9f51ebd-e158-4364-95c2-bedc152fc4b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" Nov 03 22:00:00.345733 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.345679 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2985780-6836-4a92-a0b9-40f04c597c63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll\" (UID: \"c2985780-6836-4a92-a0b9-40f04c597c63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" Nov 03 22:00:00.345849 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.345763 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4jcf\" (UniqueName: \"kubernetes.io/projected/c2985780-6836-4a92-a0b9-40f04c597c63-kube-api-access-b4jcf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll\" (UID: \"c2985780-6836-4a92-a0b9-40f04c597c63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" Nov 03 22:00:00.345849 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.345794 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5tp5\" (UniqueName: \"kubernetes.io/projected/be5a98fd-7d91-45cd-bb27-309c40495662-kube-api-access-m5tp5\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn\" (UID: \"be5a98fd-7d91-45cd-bb27-309c40495662\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" Nov 03 22:00:00.346006 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.345845 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5a98fd-7d91-45cd-bb27-309c40495662-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn\" (UID: \"be5a98fd-7d91-45cd-bb27-309c40495662\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" Nov 03 22:00:00.346006 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.345924 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zq79\" (UniqueName: \"kubernetes.io/projected/2d7c5507-a94f-4427-b503-248a20c75484-kube-api-access-6zq79\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v\" (UID: \"2d7c5507-a94f-4427-b503-248a20c75484\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" Nov 03 22:00:00.347956 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.347929 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v"] Nov 03 22:00:00.350555 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.350536 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\"" Nov 03 22:00:00.408272 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.408241 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm"] Nov 03 22:00:00.411329 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.411297 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll"] Nov 03 22:00:00.446832 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.446799 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7a8dfdc-4732-4c52-914a-53f510144184-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2\" (UID: \"f7a8dfdc-4732-4c52-914a-53f510144184\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" Nov 03 22:00:00.447036 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.446850 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5dpj\" (UniqueName: \"kubernetes.io/projected/f7a8dfdc-4732-4c52-914a-53f510144184-kube-api-access-x5dpj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2\" (UID: \"f7a8dfdc-4732-4c52-914a-53f510144184\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" Nov 03 22:00:00.447036 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.446886 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzvm7\" (UniqueName: \"kubernetes.io/projected/a9f51ebd-e158-4364-95c2-bedc152fc4b4-kube-api-access-wzvm7\") pod \"maas-api-key-cleanup-29895720-tkwzm\" (UID: \"a9f51ebd-e158-4364-95c2-bedc152fc4b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" Nov 03 22:00:00.447036 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.446924 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7c5507-a94f-4427-b503-248a20c75484-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v\" (UID: \"2d7c5507-a94f-4427-b503-248a20c75484\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" Nov 03 22:00:00.447036 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.446958 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f51ebd-e158-4364-95c2-bedc152fc4b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895720-tkwzm\" (UID: \"a9f51ebd-e158-4364-95c2-bedc152fc4b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" Nov 03 22:00:00.447036 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.447005 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2985780-6836-4a92-a0b9-40f04c597c63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll\" (UID: \"c2985780-6836-4a92-a0b9-40f04c597c63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" Nov 03 22:00:00.447268 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.447079 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4jcf\" (UniqueName: \"kubernetes.io/projected/c2985780-6836-4a92-a0b9-40f04c597c63-kube-api-access-b4jcf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll\" (UID: \"c2985780-6836-4a92-a0b9-40f04c597c63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" Nov 03 22:00:00.447268 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.447111 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5tp5\" (UniqueName: \"kubernetes.io/projected/be5a98fd-7d91-45cd-bb27-309c40495662-kube-api-access-m5tp5\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn\" (UID: \"be5a98fd-7d91-45cd-bb27-309c40495662\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" Nov 03 22:00:00.447268 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.447157 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5a98fd-7d91-45cd-bb27-309c40495662-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn\" (UID: \"be5a98fd-7d91-45cd-bb27-309c40495662\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" Nov 03 22:00:00.447268 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.447216 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zq79\" (UniqueName: \"kubernetes.io/projected/2d7c5507-a94f-4427-b503-248a20c75484-kube-api-access-6zq79\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v\" (UID: \"2d7c5507-a94f-4427-b503-248a20c75484\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" Nov 03 22:00:00.447698 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.447665 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7a8dfdc-4732-4c52-914a-53f510144184-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2\" (UID: \"f7a8dfdc-4732-4c52-914a-53f510144184\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" Nov 03 22:00:00.447842 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.447779 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7c5507-a94f-4427-b503-248a20c75484-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v\" (UID: \"2d7c5507-a94f-4427-b503-248a20c75484\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" Nov 03 22:00:00.448147 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.448128 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f51ebd-e158-4364-95c2-bedc152fc4b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895720-tkwzm\" (UID: \"a9f51ebd-e158-4364-95c2-bedc152fc4b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" Nov 03 22:00:00.448265 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.448247 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5a98fd-7d91-45cd-bb27-309c40495662-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn\" (UID: \"be5a98fd-7d91-45cd-bb27-309c40495662\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" Nov 03 22:00:00.448347 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.448294 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2985780-6836-4a92-a0b9-40f04c597c63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll\" (UID: \"c2985780-6836-4a92-a0b9-40f04c597c63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" Nov 03 22:00:00.510641 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.510568 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzvm7\" (UniqueName: \"kubernetes.io/projected/a9f51ebd-e158-4364-95c2-bedc152fc4b4-kube-api-access-wzvm7\") pod \"maas-api-key-cleanup-29895720-tkwzm\" (UID: \"a9f51ebd-e158-4364-95c2-bedc152fc4b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" Nov 03 22:00:00.511276 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.511251 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zq79\" (UniqueName: \"kubernetes.io/projected/2d7c5507-a94f-4427-b503-248a20c75484-kube-api-access-6zq79\") pod \"maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v\" (UID: \"2d7c5507-a94f-4427-b503-248a20c75484\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" Nov 03 22:00:00.511343 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.511251 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4jcf\" (UniqueName: \"kubernetes.io/projected/c2985780-6836-4a92-a0b9-40f04c597c63-kube-api-access-b4jcf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll\" (UID: \"c2985780-6836-4a92-a0b9-40f04c597c63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" Nov 03 22:00:00.511343 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.511322 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5dpj\" (UniqueName: \"kubernetes.io/projected/f7a8dfdc-4732-4c52-914a-53f510144184-kube-api-access-x5dpj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2\" (UID: \"f7a8dfdc-4732-4c52-914a-53f510144184\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" Nov 03 22:00:00.511428 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.511325 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5tp5\" (UniqueName: \"kubernetes.io/projected/be5a98fd-7d91-45cd-bb27-309c40495662-kube-api-access-m5tp5\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn\" (UID: \"be5a98fd-7d91-45cd-bb27-309c40495662\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" Nov 03 22:00:00.551331 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.551301 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" Nov 03 22:00:00.559864 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.559839 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" Nov 03 22:00:00.570917 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.570879 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" Nov 03 22:00:00.576558 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.576532 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" Nov 03 22:00:00.583815 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.583757 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" Nov 03 22:00:00.864311 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.864240 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 22:00:00.953255 ip-10-0-130-134 kubenswrapper[4963]: W1103 22:00:00.953221 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5a98fd_7d91_45cd_bb27_309c40495662.slice/crio-0a4ea0e63807915091deec8fc06a1badc4cd39b5aac91a4515abc3b79b2b1f97 WatchSource:0}: Error finding container 0a4ea0e63807915091deec8fc06a1badc4cd39b5aac91a4515abc3b79b2b1f97: Status 404 returned error can't find the container with id 0a4ea0e63807915091deec8fc06a1badc4cd39b5aac91a4515abc3b79b2b1f97 Nov 03 22:00:00.954165 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:00.954137 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn"] Nov 03 22:00:01.005728 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:01.005697 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll"] Nov 03 22:00:01.010434 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:01.010405 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2"] Nov 03 22:00:01.014458 ip-10-0-130-134 kubenswrapper[4963]: W1103 22:00:01.014426 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2985780_6836_4a92_a0b9_40f04c597c63.slice/crio-ab3d7d12e85a854bce787a63911e2a3858e626fbe2d48aa6de4687ae372edb99 WatchSource:0}: Error finding container ab3d7d12e85a854bce787a63911e2a3858e626fbe2d48aa6de4687ae372edb99: Status 404 returned error can't find the container with id ab3d7d12e85a854bce787a63911e2a3858e626fbe2d48aa6de4687ae372edb99 Nov 03 22:00:01.015144 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:01.015110 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm"] Nov 03 22:00:01.059573 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:01.057098 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v"] Nov 03 22:00:01.059573 ip-10-0-130-134 kubenswrapper[4963]: W1103 22:00:01.058225 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7c5507_a94f_4427_b503_248a20c75484.slice/crio-94fcf6d1d2a1b74dd054a4353b539ca8e0a292f69a97fff473458238a4bab79b WatchSource:0}: Error finding container 94fcf6d1d2a1b74dd054a4353b539ca8e0a292f69a97fff473458238a4bab79b: Status 404 returned error can't find the container with id 94fcf6d1d2a1b74dd054a4353b539ca8e0a292f69a97fff473458238a4bab79b Nov 03 22:00:01.860537 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:01.860349 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" event={"ID":"a9f51ebd-e158-4364-95c2-bedc152fc4b4","Type":"ContainerStarted","Data":"b810776a87965490235de3b2d16ad52194a107e7d807ea9369e0e7e340a699c7"} Nov 03 22:00:01.860537 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:01.860414 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" event={"ID":"a9f51ebd-e158-4364-95c2-bedc152fc4b4","Type":"ContainerStarted","Data":"5022ddcf6975860450c972f87f1209879c6d2a4e87138eba24e3110915c91115"} Nov 03 22:00:01.862222 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:01.862185 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" event={"ID":"2d7c5507-a94f-4427-b503-248a20c75484","Type":"ContainerStarted","Data":"859469750d4801d7bb0c140a675eda24ae91867a1faab583b4fd1329c719c0ab"} Nov 03 22:00:01.862376 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:01.862238 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" event={"ID":"2d7c5507-a94f-4427-b503-248a20c75484","Type":"ContainerStarted","Data":"94fcf6d1d2a1b74dd054a4353b539ca8e0a292f69a97fff473458238a4bab79b"} Nov 03 22:00:01.864115 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:01.864084 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" event={"ID":"c2985780-6836-4a92-a0b9-40f04c597c63","Type":"ContainerStarted","Data":"d03c442b008829530042b5cbf0a2fb4b9dbb9c4a4bc1e492a2d1c540c83b19fc"} Nov 03 22:00:01.864258 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:01.864136 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" event={"ID":"c2985780-6836-4a92-a0b9-40f04c597c63","Type":"ContainerStarted","Data":"ab3d7d12e85a854bce787a63911e2a3858e626fbe2d48aa6de4687ae372edb99"} Nov 03 22:00:01.865795 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:01.865768 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" event={"ID":"f7a8dfdc-4732-4c52-914a-53f510144184","Type":"ContainerStarted","Data":"41337361e80861efb5f6fc680a72a24627238c67134bcdf1f6595ada10a9fbeb"} Nov 03 22:00:01.865964 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:01.865818 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" event={"ID":"f7a8dfdc-4732-4c52-914a-53f510144184","Type":"ContainerStarted","Data":"df68d5169fecef54cbedadedcae72ee99fa462a7ede6f1d751b6e3baeea6a8bd"} Nov 03 22:00:01.867571 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:01.867541 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" event={"ID":"be5a98fd-7d91-45cd-bb27-309c40495662","Type":"ContainerStarted","Data":"52a907374c06c2d6d22f8068ff326108616037fc0469918a7269f96439c7097e"} Nov 03 22:00:01.867700 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:01.867598 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" event={"ID":"be5a98fd-7d91-45cd-bb27-309c40495662","Type":"ContainerStarted","Data":"0a4ea0e63807915091deec8fc06a1badc4cd39b5aac91a4515abc3b79b2b1f97"} Nov 03 22:00:01.981260 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:01.981197 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" podStartSLOduration=1.981177298 podStartE2EDuration="1.981177298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:01.979984845 +0000 UTC m=+4351.140057121" watchObservedRunningTime="2026-11-03 22:00:01.981177298 +0000 UTC m=+4351.141249571" Nov 03 22:00:02.102719 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.102653 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" podStartSLOduration=2.102637782 podStartE2EDuration="2.102637782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:02.1012889 +0000 UTC m=+4351.261361182" watchObservedRunningTime="2026-11-03 22:00:02.102637782 +0000 UTC m=+4351.262710091" Nov 03 22:00:02.224191 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.224132 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" podStartSLOduration=2.224117375 podStartE2EDuration="2.224117375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:02.220561623 +0000 UTC m=+4351.380633897" watchObservedRunningTime="2026-11-03 22:00:02.224117375 +0000 UTC m=+4351.384189646" Nov 03 22:00:02.348558 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.348493 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" podStartSLOduration=2.348472065 podStartE2EDuration="2.348472065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:02.339142429 +0000 UTC m=+4351.499214683" watchObservedRunningTime="2026-11-03 22:00:02.348472065 +0000 UTC m=+4351.508544340" Nov 03 22:00:02.459742 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.459688 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" podStartSLOduration=2.459674091 podStartE2EDuration="2.459674091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:02.458312776 +0000 UTC m=+4351.618385048" watchObservedRunningTime="2026-11-03 22:00:02.459674091 +0000 UTC m=+4351.619746407" Nov 03 22:00:02.643570 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.643536 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:02.876044 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.875954 4963 generic.go:356] "Generic (PLEG): container finished" podID="f7a8dfdc-4732-4c52-914a-53f510144184" containerID="41337361e80861efb5f6fc680a72a24627238c67134bcdf1f6595ada10a9fbeb" exitCode=60 Nov 03 22:00:02.876211 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.876037 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" event={"ID":"f7a8dfdc-4732-4c52-914a-53f510144184","Type":"ContainerDied","Data":"41337361e80861efb5f6fc680a72a24627238c67134bcdf1f6595ada10a9fbeb"} Nov 03 22:00:02.876391 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.876372 4963 scope.go:122] "RemoveContainer" containerID="41337361e80861efb5f6fc680a72a24627238c67134bcdf1f6595ada10a9fbeb" Nov 03 22:00:02.877763 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.877739 4963 generic.go:356] "Generic (PLEG): container finished" podID="be5a98fd-7d91-45cd-bb27-309c40495662" containerID="52a907374c06c2d6d22f8068ff326108616037fc0469918a7269f96439c7097e" exitCode=60 Nov 03 22:00:02.877870 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.877804 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" event={"ID":"be5a98fd-7d91-45cd-bb27-309c40495662","Type":"ContainerDied","Data":"52a907374c06c2d6d22f8068ff326108616037fc0469918a7269f96439c7097e"} Nov 03 22:00:02.878124 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.878105 4963 scope.go:122] "RemoveContainer" containerID="52a907374c06c2d6d22f8068ff326108616037fc0469918a7269f96439c7097e" Nov 03 22:00:02.879656 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.879551 4963 generic.go:356] "Generic (PLEG): container finished" podID="a9f51ebd-e158-4364-95c2-bedc152fc4b4" containerID="b810776a87965490235de3b2d16ad52194a107e7d807ea9369e0e7e340a699c7" exitCode=60 Nov 03 22:00:02.879656 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.879620 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" event={"ID":"a9f51ebd-e158-4364-95c2-bedc152fc4b4","Type":"ContainerDied","Data":"b810776a87965490235de3b2d16ad52194a107e7d807ea9369e0e7e340a699c7"} Nov 03 22:00:02.879890 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.879874 4963 scope.go:122] "RemoveContainer" containerID="b810776a87965490235de3b2d16ad52194a107e7d807ea9369e0e7e340a699c7" Nov 03 22:00:02.881297 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.881281 4963 generic.go:356] "Generic (PLEG): container finished" podID="2d7c5507-a94f-4427-b503-248a20c75484" containerID="859469750d4801d7bb0c140a675eda24ae91867a1faab583b4fd1329c719c0ab" exitCode=60 Nov 03 22:00:02.881384 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.881370 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" event={"ID":"2d7c5507-a94f-4427-b503-248a20c75484","Type":"ContainerDied","Data":"859469750d4801d7bb0c140a675eda24ae91867a1faab583b4fd1329c719c0ab"} Nov 03 22:00:02.881642 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.881628 4963 scope.go:122] "RemoveContainer" containerID="859469750d4801d7bb0c140a675eda24ae91867a1faab583b4fd1329c719c0ab" Nov 03 22:00:02.883259 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.883238 4963 generic.go:356] "Generic (PLEG): container finished" podID="c2985780-6836-4a92-a0b9-40f04c597c63" containerID="d03c442b008829530042b5cbf0a2fb4b9dbb9c4a4bc1e492a2d1c540c83b19fc" exitCode=60 Nov 03 22:00:02.883364 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.883300 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" event={"ID":"c2985780-6836-4a92-a0b9-40f04c597c63","Type":"ContainerDied","Data":"d03c442b008829530042b5cbf0a2fb4b9dbb9c4a4bc1e492a2d1c540c83b19fc"} Nov 03 22:00:02.883586 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:02.883568 4963 scope.go:122] "RemoveContainer" containerID="d03c442b008829530042b5cbf0a2fb4b9dbb9c4a4bc1e492a2d1c540c83b19fc" Nov 03 22:00:03.894793 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:03.894763 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" event={"ID":"a9f51ebd-e158-4364-95c2-bedc152fc4b4","Type":"ContainerStarted","Data":"ef5910881662a69549696ab0bd24160f241f69130aeea41699e48408fdf11c19"} Nov 03 22:00:03.895230 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:03.895133 4963 scope.go:122] "RemoveContainer" containerID="ef5910881662a69549696ab0bd24160f241f69130aeea41699e48408fdf11c19" Nov 03 22:00:03.895455 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:03.895433 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895720-tkwzm_odh-ai-gateway-infra(a9f51ebd-e158-4364-95c2-bedc152fc4b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" podUID="a9f51ebd-e158-4364-95c2-bedc152fc4b4" Nov 03 22:00:03.896754 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:03.896731 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" event={"ID":"2d7c5507-a94f-4427-b503-248a20c75484","Type":"ContainerStarted","Data":"14737ed29da4a3a2bba097093cf49051aef376f4d5bf8c9b5d829b2be17ffbc4"} Nov 03 22:00:03.898772 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:03.898745 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" event={"ID":"c2985780-6836-4a92-a0b9-40f04c597c63","Type":"ContainerStarted","Data":"8d84c379bdcddaa12c7e83918a703ba8baed40bbbf5603982abfe3fea05bfcd5"} Nov 03 22:00:03.900476 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:03.900456 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" event={"ID":"f7a8dfdc-4732-4c52-914a-53f510144184","Type":"ContainerStarted","Data":"df4359194b9d2e60451894a5d659ff3c03ec3aef4eecbe58e220ecce0f4aa9f0"} Nov 03 22:00:03.902228 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:03.902207 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" event={"ID":"be5a98fd-7d91-45cd-bb27-309c40495662","Type":"ContainerStarted","Data":"d0857dca28e4c92131613e24d339c33ae0dc00bf7f908c5061a515d0d35ebef8"} Nov 03 22:00:04.910297 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.910260 4963 generic.go:356] "Generic (PLEG): container finished" podID="f7a8dfdc-4732-4c52-914a-53f510144184" containerID="df4359194b9d2e60451894a5d659ff3c03ec3aef4eecbe58e220ecce0f4aa9f0" exitCode=60 Nov 03 22:00:04.910774 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.910340 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" event={"ID":"f7a8dfdc-4732-4c52-914a-53f510144184","Type":"ContainerDied","Data":"df4359194b9d2e60451894a5d659ff3c03ec3aef4eecbe58e220ecce0f4aa9f0"} Nov 03 22:00:04.910774 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.910398 4963 scope.go:122] "RemoveContainer" containerID="41337361e80861efb5f6fc680a72a24627238c67134bcdf1f6595ada10a9fbeb" Nov 03 22:00:04.910774 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.910659 4963 scope.go:122] "RemoveContainer" containerID="df4359194b9d2e60451894a5d659ff3c03ec3aef4eecbe58e220ecce0f4aa9f0" Nov 03 22:00:04.911016 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:04.910992 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2_odh-ai-gateway-infra(f7a8dfdc-4732-4c52-914a-53f510144184)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" podUID="f7a8dfdc-4732-4c52-914a-53f510144184" Nov 03 22:00:04.912648 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.912622 4963 generic.go:356] "Generic (PLEG): container finished" podID="be5a98fd-7d91-45cd-bb27-309c40495662" containerID="d0857dca28e4c92131613e24d339c33ae0dc00bf7f908c5061a515d0d35ebef8" exitCode=60 Nov 03 22:00:04.912765 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.912718 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" event={"ID":"be5a98fd-7d91-45cd-bb27-309c40495662","Type":"ContainerDied","Data":"d0857dca28e4c92131613e24d339c33ae0dc00bf7f908c5061a515d0d35ebef8"} Nov 03 22:00:04.912979 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.912946 4963 scope.go:122] "RemoveContainer" containerID="d0857dca28e4c92131613e24d339c33ae0dc00bf7f908c5061a515d0d35ebef8" Nov 03 22:00:04.913264 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:04.913242 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn_odh-ai-gateway-infra(be5a98fd-7d91-45cd-bb27-309c40495662)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" podUID="be5a98fd-7d91-45cd-bb27-309c40495662" Nov 03 22:00:04.914894 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.914868 4963 generic.go:356] "Generic (PLEG): container finished" podID="a9f51ebd-e158-4364-95c2-bedc152fc4b4" containerID="ef5910881662a69549696ab0bd24160f241f69130aeea41699e48408fdf11c19" exitCode=60 Nov 03 22:00:04.915050 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.914956 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" event={"ID":"a9f51ebd-e158-4364-95c2-bedc152fc4b4","Type":"ContainerDied","Data":"ef5910881662a69549696ab0bd24160f241f69130aeea41699e48408fdf11c19"} Nov 03 22:00:04.915253 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.915236 4963 scope.go:122] "RemoveContainer" containerID="ef5910881662a69549696ab0bd24160f241f69130aeea41699e48408fdf11c19" Nov 03 22:00:04.915516 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:04.915495 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895720-tkwzm_odh-ai-gateway-infra(a9f51ebd-e158-4364-95c2-bedc152fc4b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" podUID="a9f51ebd-e158-4364-95c2-bedc152fc4b4" Nov 03 22:00:04.916877 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.916855 4963 generic.go:356] "Generic (PLEG): container finished" podID="2d7c5507-a94f-4427-b503-248a20c75484" containerID="14737ed29da4a3a2bba097093cf49051aef376f4d5bf8c9b5d829b2be17ffbc4" exitCode=60 Nov 03 22:00:04.917005 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.916952 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" event={"ID":"2d7c5507-a94f-4427-b503-248a20c75484","Type":"ContainerDied","Data":"14737ed29da4a3a2bba097093cf49051aef376f4d5bf8c9b5d829b2be17ffbc4"} Nov 03 22:00:04.917222 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.917205 4963 scope.go:122] "RemoveContainer" containerID="14737ed29da4a3a2bba097093cf49051aef376f4d5bf8c9b5d829b2be17ffbc4" Nov 03 22:00:04.917483 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:04.917461 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v_odh-ai-gateway-infra(2d7c5507-a94f-4427-b503-248a20c75484)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" podUID="2d7c5507-a94f-4427-b503-248a20c75484" Nov 03 22:00:04.919051 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.919028 4963 generic.go:356] "Generic (PLEG): container finished" podID="c2985780-6836-4a92-a0b9-40f04c597c63" containerID="8d84c379bdcddaa12c7e83918a703ba8baed40bbbf5603982abfe3fea05bfcd5" exitCode=60 Nov 03 22:00:04.919152 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.919115 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" event={"ID":"c2985780-6836-4a92-a0b9-40f04c597c63","Type":"ContainerDied","Data":"8d84c379bdcddaa12c7e83918a703ba8baed40bbbf5603982abfe3fea05bfcd5"} Nov 03 22:00:04.919361 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.919347 4963 scope.go:122] "RemoveContainer" containerID="8d84c379bdcddaa12c7e83918a703ba8baed40bbbf5603982abfe3fea05bfcd5" Nov 03 22:00:04.919580 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:04.919562 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll_odh-ai-gateway-infra(c2985780-6836-4a92-a0b9-40f04c597c63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" podUID="c2985780-6836-4a92-a0b9-40f04c597c63" Nov 03 22:00:04.924263 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.924237 4963 scope.go:122] "RemoveContainer" containerID="52a907374c06c2d6d22f8068ff326108616037fc0469918a7269f96439c7097e" Nov 03 22:00:04.938253 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.938231 4963 scope.go:122] "RemoveContainer" containerID="b810776a87965490235de3b2d16ad52194a107e7d807ea9369e0e7e340a699c7" Nov 03 22:00:04.952710 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.952684 4963 scope.go:122] "RemoveContainer" containerID="859469750d4801d7bb0c140a675eda24ae91867a1faab583b4fd1329c719c0ab" Nov 03 22:00:04.967475 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:04.967453 4963 scope.go:122] "RemoveContainer" containerID="d03c442b008829530042b5cbf0a2fb4b9dbb9c4a4bc1e492a2d1c540c83b19fc" Nov 03 22:00:05.929068 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:05.929006 4963 scope.go:122] "RemoveContainer" containerID="ef5910881662a69549696ab0bd24160f241f69130aeea41699e48408fdf11c19" Nov 03 22:00:05.929550 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:05.929371 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895720-tkwzm_odh-ai-gateway-infra(a9f51ebd-e158-4364-95c2-bedc152fc4b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" podUID="a9f51ebd-e158-4364-95c2-bedc152fc4b4" Nov 03 22:00:05.930710 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:05.930689 4963 scope.go:122] "RemoveContainer" containerID="14737ed29da4a3a2bba097093cf49051aef376f4d5bf8c9b5d829b2be17ffbc4" Nov 03 22:00:05.931024 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:05.930998 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v_odh-ai-gateway-infra(2d7c5507-a94f-4427-b503-248a20c75484)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" podUID="2d7c5507-a94f-4427-b503-248a20c75484" Nov 03 22:00:05.932268 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:05.932248 4963 scope.go:122] "RemoveContainer" containerID="8d84c379bdcddaa12c7e83918a703ba8baed40bbbf5603982abfe3fea05bfcd5" Nov 03 22:00:05.932560 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:05.932539 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll_odh-ai-gateway-infra(c2985780-6836-4a92-a0b9-40f04c597c63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" podUID="c2985780-6836-4a92-a0b9-40f04c597c63" Nov 03 22:00:05.934033 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:05.934012 4963 scope.go:122] "RemoveContainer" containerID="df4359194b9d2e60451894a5d659ff3c03ec3aef4eecbe58e220ecce0f4aa9f0" Nov 03 22:00:05.934285 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:05.934257 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2_odh-ai-gateway-infra(f7a8dfdc-4732-4c52-914a-53f510144184)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" podUID="f7a8dfdc-4732-4c52-914a-53f510144184" Nov 03 22:00:05.935636 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:05.935615 4963 scope.go:122] "RemoveContainer" containerID="d0857dca28e4c92131613e24d339c33ae0dc00bf7f908c5061a515d0d35ebef8" Nov 03 22:00:05.935870 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:05.935849 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn_odh-ai-gateway-infra(be5a98fd-7d91-45cd-bb27-309c40495662)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" podUID="be5a98fd-7d91-45cd-bb27-309c40495662" Nov 03 22:00:07.147063 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:07.147015 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:08.391677 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:08.388854 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:11.640939 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:11.640842 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:12.529052 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:12.529016 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:00:15.960259 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:15.960219 4963 scope.go:122] "RemoveContainer" containerID="d0857dca28e4c92131613e24d339c33ae0dc00bf7f908c5061a515d0d35ebef8" Nov 03 22:00:15.960769 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:15.960329 4963 scope.go:122] "RemoveContainer" containerID="df4359194b9d2e60451894a5d659ff3c03ec3aef4eecbe58e220ecce0f4aa9f0" Nov 03 22:00:15.960769 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:15.960430 4963 scope.go:122] "RemoveContainer" containerID="14737ed29da4a3a2bba097093cf49051aef376f4d5bf8c9b5d829b2be17ffbc4" Nov 03 22:00:16.202379 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:16.202353 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:16.960936 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:16.960880 4963 scope.go:122] "RemoveContainer" containerID="ef5910881662a69549696ab0bd24160f241f69130aeea41699e48408fdf11c19" Nov 03 22:00:17.014390 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:17.014353 4963 generic.go:356] "Generic (PLEG): container finished" podID="2d7c5507-a94f-4427-b503-248a20c75484" containerID="f1e366c74a3a3b31042b04457d74c306aff8abdf4f6190ee47254e121cd1f9ea" exitCode=60 Nov 03 22:00:17.014529 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:17.014423 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" event={"ID":"2d7c5507-a94f-4427-b503-248a20c75484","Type":"ContainerDied","Data":"f1e366c74a3a3b31042b04457d74c306aff8abdf4f6190ee47254e121cd1f9ea"} Nov 03 22:00:17.014529 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:17.014485 4963 scope.go:122] "RemoveContainer" containerID="14737ed29da4a3a2bba097093cf49051aef376f4d5bf8c9b5d829b2be17ffbc4" Nov 03 22:00:17.014894 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:17.014873 4963 scope.go:122] "RemoveContainer" containerID="f1e366c74a3a3b31042b04457d74c306aff8abdf4f6190ee47254e121cd1f9ea" Nov 03 22:00:17.015222 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:17.015200 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v_odh-ai-gateway-infra(2d7c5507-a94f-4427-b503-248a20c75484)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" podUID="2d7c5507-a94f-4427-b503-248a20c75484" Nov 03 22:00:17.016763 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:17.016742 4963 generic.go:356] "Generic (PLEG): container finished" podID="f7a8dfdc-4732-4c52-914a-53f510144184" containerID="e9aae0f16e5543961f3f43f42917c8e559a30aa959c6ff8148c700ec16c2c3ae" exitCode=60 Nov 03 22:00:17.016852 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:17.016811 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" event={"ID":"f7a8dfdc-4732-4c52-914a-53f510144184","Type":"ContainerDied","Data":"e9aae0f16e5543961f3f43f42917c8e559a30aa959c6ff8148c700ec16c2c3ae"} Nov 03 22:00:17.017198 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:17.017182 4963 scope.go:122] "RemoveContainer" containerID="e9aae0f16e5543961f3f43f42917c8e559a30aa959c6ff8148c700ec16c2c3ae" Nov 03 22:00:17.017480 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:17.017456 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2_odh-ai-gateway-infra(f7a8dfdc-4732-4c52-914a-53f510144184)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" podUID="f7a8dfdc-4732-4c52-914a-53f510144184" Nov 03 22:00:17.018617 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:17.018600 4963 generic.go:356] "Generic (PLEG): container finished" podID="be5a98fd-7d91-45cd-bb27-309c40495662" containerID="54e9b01c1dad5d655498e0b0fabf0dd9f005420556c5d847013aa3b2d3cb3182" exitCode=60 Nov 03 22:00:17.018696 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:17.018649 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" event={"ID":"be5a98fd-7d91-45cd-bb27-309c40495662","Type":"ContainerDied","Data":"54e9b01c1dad5d655498e0b0fabf0dd9f005420556c5d847013aa3b2d3cb3182"} Nov 03 22:00:17.018989 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:17.018976 4963 scope.go:122] "RemoveContainer" containerID="54e9b01c1dad5d655498e0b0fabf0dd9f005420556c5d847013aa3b2d3cb3182" Nov 03 22:00:17.019222 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:17.019206 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn_odh-ai-gateway-infra(be5a98fd-7d91-45cd-bb27-309c40495662)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" podUID="be5a98fd-7d91-45cd-bb27-309c40495662" Nov 03 22:00:17.033172 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:17.033147 4963 scope.go:122] "RemoveContainer" containerID="df4359194b9d2e60451894a5d659ff3c03ec3aef4eecbe58e220ecce0f4aa9f0" Nov 03 22:00:17.047087 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:17.047062 4963 scope.go:122] "RemoveContainer" containerID="d0857dca28e4c92131613e24d339c33ae0dc00bf7f908c5061a515d0d35ebef8" Nov 03 22:00:17.960357 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:17.960327 4963 scope.go:122] "RemoveContainer" containerID="8d84c379bdcddaa12c7e83918a703ba8baed40bbbf5603982abfe3fea05bfcd5" Nov 03 22:00:18.031919 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.031871 4963 generic.go:356] "Generic (PLEG): container finished" podID="a9f51ebd-e158-4364-95c2-bedc152fc4b4" containerID="5c3d7beb271aa25c7de7202ac7ae74fd25855fcaa997b41c2397dda12c2946ce" exitCode=60 Nov 03 22:00:18.032401 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.031947 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" event={"ID":"a9f51ebd-e158-4364-95c2-bedc152fc4b4","Type":"ContainerDied","Data":"5c3d7beb271aa25c7de7202ac7ae74fd25855fcaa997b41c2397dda12c2946ce"} Nov 03 22:00:18.032401 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.032014 4963 scope.go:122] "RemoveContainer" containerID="ef5910881662a69549696ab0bd24160f241f69130aeea41699e48408fdf11c19" Nov 03 22:00:18.032520 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.032434 4963 scope.go:122] "RemoveContainer" containerID="5c3d7beb271aa25c7de7202ac7ae74fd25855fcaa997b41c2397dda12c2946ce" Nov 03 22:00:18.032793 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:18.032765 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895720-tkwzm_odh-ai-gateway-infra(a9f51ebd-e158-4364-95c2-bedc152fc4b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" podUID="a9f51ebd-e158-4364-95c2-bedc152fc4b4" Nov 03 22:00:18.136034 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.135995 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v"] Nov 03 22:00:18.245649 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.245595 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn"] Nov 03 22:00:18.351804 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.351517 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" Nov 03 22:00:18.355844 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.355726 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2"] Nov 03 22:00:18.459078 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.458987 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" Nov 03 22:00:18.461700 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.461673 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d7c5507-a94f-4427-b503-248a20c75484-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7c5507-a94f-4427-b503-248a20c75484-maas-api-service-ca\") pod \"2d7c5507-a94f-4427-b503-248a20c75484\" (UID: \"2d7c5507-a94f-4427-b503-248a20c75484\") " Nov 03 22:00:18.461868 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.461750 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7c5507-a94f-4427-b503-248a20c75484-kube-api-access-6zq79\" (UniqueName: \"kubernetes.io/projected/2d7c5507-a94f-4427-b503-248a20c75484-kube-api-access-6zq79\") pod \"2d7c5507-a94f-4427-b503-248a20c75484\" (UID: \"2d7c5507-a94f-4427-b503-248a20c75484\") " Nov 03 22:00:18.462792 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.462748 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7c5507-a94f-4427-b503-248a20c75484-maas-api-service-ca" pod "2d7c5507-a94f-4427-b503-248a20c75484" (UID: "2d7c5507-a94f-4427-b503-248a20c75484"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:18.468528 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.468485 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7c5507-a94f-4427-b503-248a20c75484-kube-api-access-6zq79" pod "2d7c5507-a94f-4427-b503-248a20c75484" (UID: "2d7c5507-a94f-4427-b503-248a20c75484"). InnerVolumeSpecName "kube-api-access-6zq79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:18.540124 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.540091 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" Nov 03 22:00:18.563532 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.563441 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be5a98fd-7d91-45cd-bb27-309c40495662-kube-api-access-m5tp5\" (UniqueName: \"kubernetes.io/projected/be5a98fd-7d91-45cd-bb27-309c40495662-kube-api-access-m5tp5\") pod \"be5a98fd-7d91-45cd-bb27-309c40495662\" (UID: \"be5a98fd-7d91-45cd-bb27-309c40495662\") " Nov 03 22:00:18.563696 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.563556 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be5a98fd-7d91-45cd-bb27-309c40495662-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5a98fd-7d91-45cd-bb27-309c40495662-maas-api-service-ca\") pod \"be5a98fd-7d91-45cd-bb27-309c40495662\" (UID: \"be5a98fd-7d91-45cd-bb27-309c40495662\") " Nov 03 22:00:18.564013 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.563968 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be5a98fd-7d91-45cd-bb27-309c40495662-maas-api-service-ca" pod "be5a98fd-7d91-45cd-bb27-309c40495662" (UID: "be5a98fd-7d91-45cd-bb27-309c40495662"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:18.564175 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.564158 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7c5507-a94f-4427-b503-248a20c75484-maas-api-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:18.564237 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.564177 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zq79\" (UniqueName: \"kubernetes.io/projected/2d7c5507-a94f-4427-b503-248a20c75484-kube-api-access-6zq79\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:18.564237 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.564187 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5a98fd-7d91-45cd-bb27-309c40495662-maas-api-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:18.565830 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.565800 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5a98fd-7d91-45cd-bb27-309c40495662-kube-api-access-m5tp5" pod "be5a98fd-7d91-45cd-bb27-309c40495662" (UID: "be5a98fd-7d91-45cd-bb27-309c40495662"). InnerVolumeSpecName "kube-api-access-m5tp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:18.664584 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.664548 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7a8dfdc-4732-4c52-914a-53f510144184-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7a8dfdc-4732-4c52-914a-53f510144184-maas-api-service-ca\") pod \"f7a8dfdc-4732-4c52-914a-53f510144184\" (UID: \"f7a8dfdc-4732-4c52-914a-53f510144184\") " Nov 03 22:00:18.664767 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.664655 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a8dfdc-4732-4c52-914a-53f510144184-kube-api-access-x5dpj\" (UniqueName: \"kubernetes.io/projected/f7a8dfdc-4732-4c52-914a-53f510144184-kube-api-access-x5dpj\") pod \"f7a8dfdc-4732-4c52-914a-53f510144184\" (UID: \"f7a8dfdc-4732-4c52-914a-53f510144184\") " Nov 03 22:00:18.664980 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.664955 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a8dfdc-4732-4c52-914a-53f510144184-maas-api-service-ca" pod "f7a8dfdc-4732-4c52-914a-53f510144184" (UID: "f7a8dfdc-4732-4c52-914a-53f510144184"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:18.665220 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.665203 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7a8dfdc-4732-4c52-914a-53f510144184-maas-api-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:18.665277 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.665222 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5tp5\" (UniqueName: \"kubernetes.io/projected/be5a98fd-7d91-45cd-bb27-309c40495662-kube-api-access-m5tp5\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:18.667042 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.667009 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a8dfdc-4732-4c52-914a-53f510144184-kube-api-access-x5dpj" pod "f7a8dfdc-4732-4c52-914a-53f510144184" (UID: "f7a8dfdc-4732-4c52-914a-53f510144184"). InnerVolumeSpecName "kube-api-access-x5dpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:18.766641 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:18.766606 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5dpj\" (UniqueName: \"kubernetes.io/projected/f7a8dfdc-4732-4c52-914a-53f510144184-kube-api-access-x5dpj\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:19.043040 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.043001 4963 generic.go:356] "Generic (PLEG): container finished" podID="c2985780-6836-4a92-a0b9-40f04c597c63" containerID="358672010ae7a9a136a1355bb054684ba7fe4e7e8b1e45e5379a075b47f4801d" exitCode=60 Nov 03 22:00:19.043553 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.043081 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" event={"ID":"c2985780-6836-4a92-a0b9-40f04c597c63","Type":"ContainerDied","Data":"358672010ae7a9a136a1355bb054684ba7fe4e7e8b1e45e5379a075b47f4801d"} Nov 03 22:00:19.043553 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.043158 4963 scope.go:122] "RemoveContainer" containerID="8d84c379bdcddaa12c7e83918a703ba8baed40bbbf5603982abfe3fea05bfcd5" Nov 03 22:00:19.043770 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.043571 4963 scope.go:122] "RemoveContainer" containerID="358672010ae7a9a136a1355bb054684ba7fe4e7e8b1e45e5379a075b47f4801d" Nov 03 22:00:19.043886 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:19.043865 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll_odh-ai-gateway-infra(c2985780-6836-4a92-a0b9-40f04c597c63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" podUID="c2985780-6836-4a92-a0b9-40f04c597c63" Nov 03 22:00:19.044896 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.044872 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" Nov 03 22:00:19.044896 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.044892 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2" event={"ID":"f7a8dfdc-4732-4c52-914a-53f510144184","Type":"ContainerDied","Data":"df68d5169fecef54cbedadedcae72ee99fa462a7ede6f1d751b6e3baeea6a8bd"} Nov 03 22:00:19.046545 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.046525 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" Nov 03 22:00:19.046669 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.046561 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn" event={"ID":"be5a98fd-7d91-45cd-bb27-309c40495662","Type":"ContainerDied","Data":"0a4ea0e63807915091deec8fc06a1badc4cd39b5aac91a4515abc3b79b2b1f97"} Nov 03 22:00:19.050530 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.050502 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" event={"ID":"2d7c5507-a94f-4427-b503-248a20c75484","Type":"ContainerDied","Data":"94fcf6d1d2a1b74dd054a4353b539ca8e0a292f69a97fff473458238a4bab79b"} Nov 03 22:00:19.050530 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.050525 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v" Nov 03 22:00:19.058188 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.058167 4963 scope.go:122] "RemoveContainer" containerID="e9aae0f16e5543961f3f43f42917c8e559a30aa959c6ff8148c700ec16c2c3ae" Nov 03 22:00:19.068879 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.068849 4963 scope.go:122] "RemoveContainer" containerID="54e9b01c1dad5d655498e0b0fabf0dd9f005420556c5d847013aa3b2d3cb3182" Nov 03 22:00:19.082250 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.082213 4963 scope.go:122] "RemoveContainer" containerID="f1e366c74a3a3b31042b04457d74c306aff8abdf4f6190ee47254e121cd1f9ea" Nov 03 22:00:19.158102 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.158041 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm"] Nov 03 22:00:19.262696 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.262614 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn"] Nov 03 22:00:19.273711 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.273673 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895720-7t6qn"] Nov 03 22:00:19.298391 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.298354 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" Nov 03 22:00:19.373831 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.373791 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2"] Nov 03 22:00:19.374176 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.374151 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9f51ebd-e158-4364-95c2-bedc152fc4b4-kube-api-access-wzvm7\" (UniqueName: \"kubernetes.io/projected/a9f51ebd-e158-4364-95c2-bedc152fc4b4-kube-api-access-wzvm7\") pod \"a9f51ebd-e158-4364-95c2-bedc152fc4b4\" (UID: \"a9f51ebd-e158-4364-95c2-bedc152fc4b4\") " Nov 03 22:00:19.374282 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.374250 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9f51ebd-e158-4364-95c2-bedc152fc4b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f51ebd-e158-4364-95c2-bedc152fc4b4-maas-api-service-ca\") pod \"a9f51ebd-e158-4364-95c2-bedc152fc4b4\" (UID: \"a9f51ebd-e158-4364-95c2-bedc152fc4b4\") " Nov 03 22:00:19.374700 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.374673 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f51ebd-e158-4364-95c2-bedc152fc4b4-maas-api-service-ca" pod "a9f51ebd-e158-4364-95c2-bedc152fc4b4" (UID: "a9f51ebd-e158-4364-95c2-bedc152fc4b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:19.376868 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.376841 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f51ebd-e158-4364-95c2-bedc152fc4b4-kube-api-access-wzvm7" pod "a9f51ebd-e158-4364-95c2-bedc152fc4b4" (UID: "a9f51ebd-e158-4364-95c2-bedc152fc4b4"). InnerVolumeSpecName "kube-api-access-wzvm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:19.385100 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.385070 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895720-z6ps2"] Nov 03 22:00:19.476010 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.475972 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzvm7\" (UniqueName: \"kubernetes.io/projected/a9f51ebd-e158-4364-95c2-bedc152fc4b4-kube-api-access-wzvm7\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:19.476010 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.476002 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f51ebd-e158-4364-95c2-bedc152fc4b4-maas-api-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:19.486696 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.486658 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v"] Nov 03 22:00:19.491109 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:19.491071 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-099a5e-29895720-xfl2v"] Nov 03 22:00:20.057919 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.057885 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" Nov 03 22:00:20.058515 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.057885 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm" event={"ID":"a9f51ebd-e158-4364-95c2-bedc152fc4b4","Type":"ContainerDied","Data":"5022ddcf6975860450c972f87f1209879c6d2a4e87138eba24e3110915c91115"} Nov 03 22:00:20.058515 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.058054 4963 scope.go:122] "RemoveContainer" containerID="5c3d7beb271aa25c7de7202ac7ae74fd25855fcaa997b41c2397dda12c2946ce" Nov 03 22:00:20.163084 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.163051 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll"] Nov 03 22:00:20.218046 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.217968 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm"] Nov 03 22:00:20.231494 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.227016 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895720-tkwzm"] Nov 03 22:00:20.341659 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.341629 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" Nov 03 22:00:20.447634 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.447598 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h"] Nov 03 22:00:20.448232 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448209 4963 cpu_manager.go:414] "containerMap: removing container" podUID="f7a8dfdc-4732-4c52-914a-53f510144184" containerName="cleanup" Nov 03 22:00:20.448232 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448232 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a8dfdc-4732-4c52-914a-53f510144184" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448246 4963 cpu_manager.go:414] "containerMap: removing container" podUID="c2985780-6836-4a92-a0b9-40f04c597c63" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448255 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2985780-6836-4a92-a0b9-40f04c597c63" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448271 4963 cpu_manager.go:414] "containerMap: removing container" podUID="be5a98fd-7d91-45cd-bb27-309c40495662" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448280 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be5a98fd-7d91-45cd-bb27-309c40495662" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448296 4963 cpu_manager.go:414] "containerMap: removing container" podUID="be5a98fd-7d91-45cd-bb27-309c40495662" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448303 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be5a98fd-7d91-45cd-bb27-309c40495662" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448320 4963 cpu_manager.go:414] "containerMap: removing container" podUID="a9f51ebd-e158-4364-95c2-bedc152fc4b4" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448327 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f51ebd-e158-4364-95c2-bedc152fc4b4" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448361 4963 cpu_manager.go:414] "containerMap: removing container" podUID="f7a8dfdc-4732-4c52-914a-53f510144184" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448368 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a8dfdc-4732-4c52-914a-53f510144184" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448380 4963 cpu_manager.go:414] "containerMap: removing container" podUID="c2985780-6836-4a92-a0b9-40f04c597c63" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448391 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2985780-6836-4a92-a0b9-40f04c597c63" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448401 4963 cpu_manager.go:414] "containerMap: removing container" podUID="c2985780-6836-4a92-a0b9-40f04c597c63" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448409 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2985780-6836-4a92-a0b9-40f04c597c63" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448419 4963 cpu_manager.go:414] "containerMap: removing container" podUID="a9f51ebd-e158-4364-95c2-bedc152fc4b4" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448424 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f51ebd-e158-4364-95c2-bedc152fc4b4" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448433 4963 cpu_manager.go:414] "containerMap: removing container" podUID="2d7c5507-a94f-4427-b503-248a20c75484" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448438 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7c5507-a94f-4427-b503-248a20c75484" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448444 4963 cpu_manager.go:414] "containerMap: removing container" podUID="2d7c5507-a94f-4427-b503-248a20c75484" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448449 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7c5507-a94f-4427-b503-248a20c75484" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448456 4963 cpu_manager.go:414] "containerMap: removing container" podUID="a9f51ebd-e158-4364-95c2-bedc152fc4b4" containerName="cleanup" Nov 03 22:00:20.448452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448461 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f51ebd-e158-4364-95c2-bedc152fc4b4" containerName="cleanup" Nov 03 22:00:20.449609 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448585 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="be5a98fd-7d91-45cd-bb27-309c40495662" containerName="cleanup" Nov 03 22:00:20.449609 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448594 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="be5a98fd-7d91-45cd-bb27-309c40495662" containerName="cleanup" Nov 03 22:00:20.449609 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448600 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f51ebd-e158-4364-95c2-bedc152fc4b4" containerName="cleanup" Nov 03 22:00:20.449609 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448609 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a8dfdc-4732-4c52-914a-53f510144184" containerName="cleanup" Nov 03 22:00:20.449609 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448615 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a8dfdc-4732-4c52-914a-53f510144184" containerName="cleanup" Nov 03 22:00:20.449609 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448621 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2985780-6836-4a92-a0b9-40f04c597c63" containerName="cleanup" Nov 03 22:00:20.449609 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448626 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2985780-6836-4a92-a0b9-40f04c597c63" containerName="cleanup" Nov 03 22:00:20.449609 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448631 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f51ebd-e158-4364-95c2-bedc152fc4b4" containerName="cleanup" Nov 03 22:00:20.449609 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448636 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7c5507-a94f-4427-b503-248a20c75484" containerName="cleanup" Nov 03 22:00:20.449609 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448642 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7c5507-a94f-4427-b503-248a20c75484" containerName="cleanup" Nov 03 22:00:20.449609 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448648 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f51ebd-e158-4364-95c2-bedc152fc4b4" containerName="cleanup" Nov 03 22:00:20.449609 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.448655 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7c5507-a94f-4427-b503-248a20c75484" containerName="cleanup" Nov 03 22:00:20.452029 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.452007 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:00:20.489076 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.489047 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2985780-6836-4a92-a0b9-40f04c597c63-kube-api-access-b4jcf\" (UniqueName: \"kubernetes.io/projected/c2985780-6836-4a92-a0b9-40f04c597c63-kube-api-access-b4jcf\") pod \"c2985780-6836-4a92-a0b9-40f04c597c63\" (UID: \"c2985780-6836-4a92-a0b9-40f04c597c63\") " Nov 03 22:00:20.489273 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.489175 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2985780-6836-4a92-a0b9-40f04c597c63-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2985780-6836-4a92-a0b9-40f04c597c63-maas-api-service-ca\") pod \"c2985780-6836-4a92-a0b9-40f04c597c63\" (UID: \"c2985780-6836-4a92-a0b9-40f04c597c63\") " Nov 03 22:00:20.489560 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.489534 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2985780-6836-4a92-a0b9-40f04c597c63-maas-api-service-ca" pod "c2985780-6836-4a92-a0b9-40f04c597c63" (UID: "c2985780-6836-4a92-a0b9-40f04c597c63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:20.491667 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.491635 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2985780-6836-4a92-a0b9-40f04c597c63-kube-api-access-b4jcf" pod "c2985780-6836-4a92-a0b9-40f04c597c63" (UID: "c2985780-6836-4a92-a0b9-40f04c597c63"). InnerVolumeSpecName "kube-api-access-b4jcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:20.504052 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.504023 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-85db4aad-serving-cert\"" Nov 03 22:00:20.508126 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.508092 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h"] Nov 03 22:00:20.591179 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.591094 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-openshift-service-ca\") pod \"maas-api-e2e-mt-85db4aad-7586748c79-h9l4h\" (UID: \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:00:20.591179 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.591153 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-metrics-certs\") pod \"maas-api-e2e-mt-85db4aad-7586748c79-h9l4h\" (UID: \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:00:20.591386 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.591272 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf6s2\" (UniqueName: \"kubernetes.io/projected/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-kube-api-access-rf6s2\") pod \"maas-api-e2e-mt-85db4aad-7586748c79-h9l4h\" (UID: \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:00:20.591386 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.591329 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-maas-api-tls\") pod \"maas-api-e2e-mt-85db4aad-7586748c79-h9l4h\" (UID: \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:00:20.591554 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.591537 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4jcf\" (UniqueName: \"kubernetes.io/projected/c2985780-6836-4a92-a0b9-40f04c597c63-kube-api-access-b4jcf\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:20.591592 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.591556 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2985780-6836-4a92-a0b9-40f04c597c63-maas-api-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:00:20.685009 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.684972 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:20.693136 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.693099 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-openshift-service-ca\") pod \"maas-api-e2e-mt-85db4aad-7586748c79-h9l4h\" (UID: \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:00:20.693322 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.693152 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-metrics-certs\") pod \"maas-api-e2e-mt-85db4aad-7586748c79-h9l4h\" (UID: \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:00:20.693322 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.693193 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf6s2\" (UniqueName: \"kubernetes.io/projected/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-kube-api-access-rf6s2\") pod \"maas-api-e2e-mt-85db4aad-7586748c79-h9l4h\" (UID: \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:00:20.693322 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.693227 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-maas-api-tls\") pod \"maas-api-e2e-mt-85db4aad-7586748c79-h9l4h\" (UID: \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:00:20.693455 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:20.693359 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-serving-cert: secret "maas-api-e2e-mt-85db4aad-serving-cert" not found Nov 03 22:00:20.693455 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:20.693442 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-maas-api-tls podName:4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a nodeName:}" failed. No retries permitted until 2026-11-03 22:00:21.193412823 +0000 UTC m=+4370.353485086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-maas-api-tls") pod "maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" (UID: "4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a") : secret "maas-api-e2e-mt-85db4aad-serving-cert" not found Nov 03 22:00:20.694035 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.694012 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-openshift-service-ca\") pod \"maas-api-e2e-mt-85db4aad-7586748c79-h9l4h\" (UID: \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:00:20.696024 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.695996 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-metrics-certs\") pod \"maas-api-e2e-mt-85db4aad-7586748c79-h9l4h\" (UID: \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:00:20.752137 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.752096 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf6s2\" (UniqueName: \"kubernetes.io/projected/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-kube-api-access-rf6s2\") pod \"maas-api-e2e-mt-85db4aad-7586748c79-h9l4h\" (UID: \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:00:20.873683 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.873588 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:20.967378 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.967345 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d7c5507-a94f-4427-b503-248a20c75484" path="/var/lib/kubelet/pods/2d7c5507-a94f-4427-b503-248a20c75484/volumes" Nov 03 22:00:20.967799 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.967785 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9f51ebd-e158-4364-95c2-bedc152fc4b4" path="/var/lib/kubelet/pods/a9f51ebd-e158-4364-95c2-bedc152fc4b4/volumes" Nov 03 22:00:20.968182 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.968170 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be5a98fd-7d91-45cd-bb27-309c40495662" path="/var/lib/kubelet/pods/be5a98fd-7d91-45cd-bb27-309c40495662/volumes" Nov 03 22:00:20.968835 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:20.968821 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7a8dfdc-4732-4c52-914a-53f510144184" path="/var/lib/kubelet/pods/f7a8dfdc-4732-4c52-914a-53f510144184/volumes" Nov 03 22:00:21.056824 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:21.056790 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:21.071571 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:21.071538 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" Nov 03 22:00:21.072065 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:21.071574 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll" event={"ID":"c2985780-6836-4a92-a0b9-40f04c597c63","Type":"ContainerDied","Data":"ab3d7d12e85a854bce787a63911e2a3858e626fbe2d48aa6de4687ae372edb99"} Nov 03 22:00:21.072065 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:21.071659 4963 scope.go:122] "RemoveContainer" containerID="358672010ae7a9a136a1355bb054684ba7fe4e7e8b1e45e5379a075b47f4801d" Nov 03 22:00:21.199853 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:21.199814 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-maas-api-tls\") pod \"maas-api-e2e-mt-85db4aad-7586748c79-h9l4h\" (UID: \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:00:21.202359 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:21.202332 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-maas-api-tls\") pod \"maas-api-e2e-mt-85db4aad-7586748c79-h9l4h\" (UID: \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:00:21.230154 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:21.230110 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll"] Nov 03 22:00:21.239248 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:21.239212 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895720-zptll"] Nov 03 22:00:21.370458 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:21.370417 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:00:21.803781 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:21.803744 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h"] Nov 03 22:00:21.809670 ip-10-0-130-134 kubenswrapper[4963]: W1103 22:00:21.809638 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9cd8ca_bd80_40fb_b420_229f3e6fcd2a.slice/crio-3a4ff940d995252f6ea0d554a818e9a385c7155efa89be47b3307e65288c3370 WatchSource:0}: Error finding container 3a4ff940d995252f6ea0d554a818e9a385c7155efa89be47b3307e65288c3370: Status 404 returned error can't find the container with id 3a4ff940d995252f6ea0d554a818e9a385c7155efa89be47b3307e65288c3370 Nov 03 22:00:21.811406 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:21.811386 4963 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:00:22.080040 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:22.079952 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" event={"ID":"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a","Type":"ContainerStarted","Data":"3a4ff940d995252f6ea0d554a818e9a385c7155efa89be47b3307e65288c3370"} Nov 03 22:00:22.967940 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:22.967895 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2985780-6836-4a92-a0b9-40f04c597c63" path="/var/lib/kubelet/pods/c2985780-6836-4a92-a0b9-40f04c597c63/volumes" Nov 03 22:00:23.087529 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:23.087493 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" event={"ID":"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a","Type":"ContainerStarted","Data":"1e40c4a14875311e74bbbb1719d0a5e828f7f5568e287fc2f9c405d922b74896"} Nov 03 22:00:23.087971 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:23.087562 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:00:23.204963 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:23.204895 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" podStartSLOduration=2.128387794 podStartE2EDuration="3.2048783s" totalImagesPullingTime="1.076490506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:23.202048081 +0000 UTC m=+4372.362120366" watchObservedRunningTime="2026-11-03 22:00:23.2048783 +0000 UTC m=+4372.364950575" Nov 03 22:00:25.345234 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:25.345198 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:29.099835 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:29.099800 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:00:29.911079 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:29.911034 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:34.406449 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:34.406413 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:38.933170 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:38.933132 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:42.532555 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:00:42.532512 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:00:43.524214 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:43.524171 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:45.766240 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:45.766204 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:47.077588 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:47.077551 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:47.305791 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:47.305759 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:48.110398 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:48.110360 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:52.594511 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:52.594475 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:00:57.133403 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:00:57.133363 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:00.431936 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:00.431871 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:00.619306 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:00.619268 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:01.710466 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:01.710427 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:06.300587 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:06.300549 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:10.779996 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:10.779961 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:12.529283 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:01:12.529236 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:01:15.270291 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:15.270257 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:19.728015 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:19.727978 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:24.257780 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:24.257593 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:28.703621 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:28.703586 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:33.332790 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:33.332755 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:35.196955 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:35.196895 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:35.385493 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:35.385452 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:35.824162 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:35.824126 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:38.105321 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:38.105287 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:42.223794 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:42.223712 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:42.530697 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:01:42.530610 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:01:42.667243 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:42.667206 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:46.119060 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:46.119000 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:47.198588 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:47.198554 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:51.644481 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:51.644446 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:56.296395 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:56.296336 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:57.616341 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:57.616288 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7478-20c61c"] Nov 03 22:01:57.616871 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:57.616856 4963 cpu_manager.go:414] "containerMap: removing container" podUID="2d7c5507-a94f-4427-b503-248a20c75484" containerName="cleanup" Nov 03 22:01:57.616963 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:57.616876 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7c5507-a94f-4427-b503-248a20c75484" containerName="cleanup" Nov 03 22:01:57.616963 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:57.616930 4963 cpu_manager.go:414] "containerMap: removing container" podUID="f7a8dfdc-4732-4c52-914a-53f510144184" containerName="cleanup" Nov 03 22:01:57.616963 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:57.616939 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a8dfdc-4732-4c52-914a-53f510144184" containerName="cleanup" Nov 03 22:01:57.616963 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:57.616961 4963 cpu_manager.go:414] "containerMap: removing container" podUID="be5a98fd-7d91-45cd-bb27-309c40495662" containerName="cleanup" Nov 03 22:01:57.617169 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:57.616968 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be5a98fd-7d91-45cd-bb27-309c40495662" containerName="cleanup" Nov 03 22:01:57.617169 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:57.617102 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="be5a98fd-7d91-45cd-bb27-309c40495662" containerName="cleanup" Nov 03 22:01:57.617169 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:57.617122 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a8dfdc-4732-4c52-914a-53f510144184" containerName="cleanup" Nov 03 22:01:57.617169 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:57.617130 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2985780-6836-4a92-a0b9-40f04c597c63" containerName="cleanup" Nov 03 22:01:57.620641 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:57.620620 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-20c61c" Nov 03 22:01:57.670658 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:57.670622 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-20c61c"] Nov 03 22:01:57.713879 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:57.713848 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7vxf5\"" Nov 03 22:01:57.735371 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:57.735335 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjwqp\" (UniqueName: \"kubernetes.io/projected/2ae1daa1-61f5-4fae-a66b-e87d1895a8b5-kube-api-access-sjwqp\") pod \"test-curl-7478-20c61c\" (UID: \"2ae1daa1-61f5-4fae-a66b-e87d1895a8b5\") " pod="opendatahub/test-curl-7478-20c61c" Nov 03 22:01:57.836417 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:57.836377 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjwqp\" (UniqueName: \"kubernetes.io/projected/2ae1daa1-61f5-4fae-a66b-e87d1895a8b5-kube-api-access-sjwqp\") pod \"test-curl-7478-20c61c\" (UID: \"2ae1daa1-61f5-4fae-a66b-e87d1895a8b5\") " pod="opendatahub/test-curl-7478-20c61c" Nov 03 22:01:57.894617 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:57.894539 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjwqp\" (UniqueName: \"kubernetes.io/projected/2ae1daa1-61f5-4fae-a66b-e87d1895a8b5-kube-api-access-sjwqp\") pod \"test-curl-7478-20c61c\" (UID: \"2ae1daa1-61f5-4fae-a66b-e87d1895a8b5\") " pod="opendatahub/test-curl-7478-20c61c" Nov 03 22:01:57.934426 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:57.934392 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-20c61c" Nov 03 22:01:58.011077 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:58.011000 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:01:58.310476 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:58.310423 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-20c61c"] Nov 03 22:01:58.315965 ip-10-0-130-134 kubenswrapper[4963]: W1103 22:01:58.315933 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ae1daa1_61f5_4fae_a66b_e87d1895a8b5.slice/crio-4549469b33c4165fbd1c70a62322293dbb283747dcdf0bbb07e64a16c41091a6 WatchSource:0}: Error finding container 4549469b33c4165fbd1c70a62322293dbb283747dcdf0bbb07e64a16c41091a6: Status 404 returned error can't find the container with id 4549469b33c4165fbd1c70a62322293dbb283747dcdf0bbb07e64a16c41091a6 Nov 03 22:01:58.695308 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:58.695269 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-20c61c" event={"ID":"2ae1daa1-61f5-4fae-a66b-e87d1895a8b5","Type":"ContainerStarted","Data":"832885e003de38ee5f79e08ea2e8fab4163907f48a14ed1137af092c9b2f80b9"} Nov 03 22:01:58.695842 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:58.695323 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-20c61c" event={"ID":"2ae1daa1-61f5-4fae-a66b-e87d1895a8b5","Type":"ContainerStarted","Data":"4549469b33c4165fbd1c70a62322293dbb283747dcdf0bbb07e64a16c41091a6"} Nov 03 22:01:58.804281 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:01:58.804221 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7478-20c61c" podStartSLOduration=1.804204798 podStartE2EDuration="1.804204798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:01:58.802349063 +0000 UTC m=+4467.962421352" watchObservedRunningTime="2026-11-03 22:01:58.804204798 +0000 UTC m=+4467.964277070" Nov 03 22:02:00.306015 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:00.305973 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7478-20c61c"] Nov 03 22:02:00.311094 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:00.311055 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7478-20c61c"] Nov 03 22:02:00.462970 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:00.462934 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7478-ce7155"] Nov 03 22:02:00.463613 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:00.463592 4963 cpu_manager.go:414] "containerMap: removing container" podUID="2ae1daa1-61f5-4fae-a66b-e87d1895a8b5" containerName="test-curl-7478-20c61c" Nov 03 22:02:00.463675 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:00.463616 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae1daa1-61f5-4fae-a66b-e87d1895a8b5" containerName="test-curl-7478-20c61c" Nov 03 22:02:00.463798 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:00.463784 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae1daa1-61f5-4fae-a66b-e87d1895a8b5" containerName="test-curl-7478-20c61c" Nov 03 22:02:00.467354 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:00.467335 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-ce7155" Nov 03 22:02:00.481526 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:00.481496 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8qsl\" (UniqueName: \"kubernetes.io/projected/f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42-kube-api-access-l8qsl\") pod \"test-curl-7478-ce7155\" (UID: \"f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42\") " pod="opendatahub/test-curl-7478-ce7155" Nov 03 22:02:00.517867 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:00.517835 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-ce7155"] Nov 03 22:02:00.582940 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:00.582808 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8qsl\" (UniqueName: \"kubernetes.io/projected/f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42-kube-api-access-l8qsl\") pod \"test-curl-7478-ce7155\" (UID: \"f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42\") " pod="opendatahub/test-curl-7478-ce7155" Nov 03 22:02:00.641600 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:00.641559 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8qsl\" (UniqueName: \"kubernetes.io/projected/f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42-kube-api-access-l8qsl\") pod \"test-curl-7478-ce7155\" (UID: \"f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42\") " pod="opendatahub/test-curl-7478-ce7155" Nov 03 22:02:00.707571 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:00.707477 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7478-20c61c" podUID="2ae1daa1-61f5-4fae-a66b-e87d1895a8b5" containerName="test-curl-7478-20c61c" containerID="cri-o://832885e003de38ee5f79e08ea2e8fab4163907f48a14ed1137af092c9b2f80b9" gracePeriod=2 Nov 03 22:02:00.756088 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:02:00.756038 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7478-20c61c\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="2ae1daa1-61f5-4fae-a66b-e87d1895a8b5" pod="opendatahub/test-curl-7478-20c61c" Nov 03 22:02:00.780763 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:00.780726 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-ce7155" Nov 03 22:02:00.823408 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:00.823370 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:01.012506 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:02:01.012467 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7478-20c61c\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="2ae1daa1-61f5-4fae-a66b-e87d1895a8b5" pod="opendatahub/test-curl-7478-20c61c" Nov 03 22:02:01.158336 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:01.158243 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-ce7155"] Nov 03 22:02:01.164019 ip-10-0-130-134 kubenswrapper[4963]: W1103 22:02:01.163985 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1cb515f_e056_4ccb_8e79_a2ab6dbb5b42.slice/crio-cb3d0015bbfec9ff4c8f18e7511f7bf158d5188e3bce94f76633b91c9798947e WatchSource:0}: Error finding container cb3d0015bbfec9ff4c8f18e7511f7bf158d5188e3bce94f76633b91c9798947e: Status 404 returned error can't find the container with id cb3d0015bbfec9ff4c8f18e7511f7bf158d5188e3bce94f76633b91c9798947e Nov 03 22:02:01.715330 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:01.715290 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-ce7155" event={"ID":"f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42","Type":"ContainerStarted","Data":"49e414a4a34020bb723032488dcba91060297c448fe3257f577ede791d13a9be"} Nov 03 22:02:01.715330 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:01.715336 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-ce7155" event={"ID":"f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42","Type":"ContainerStarted","Data":"cb3d0015bbfec9ff4c8f18e7511f7bf158d5188e3bce94f76633b91c9798947e"} Nov 03 22:02:01.824163 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:01.824108 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7478-ce7155" podStartSLOduration=1.824094209 podStartE2EDuration="1.824094209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:01.823184993 +0000 UTC m=+4470.983257266" watchObservedRunningTime="2026-11-03 22:02:01.824094209 +0000 UTC m=+4470.984166481" Nov 03 22:02:02.973147 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:02.973109 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-20c61c" Nov 03 22:02:03.010833 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:03.010789 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae1daa1-61f5-4fae-a66b-e87d1895a8b5-kube-api-access-sjwqp\" (UniqueName: \"kubernetes.io/projected/2ae1daa1-61f5-4fae-a66b-e87d1895a8b5-kube-api-access-sjwqp\") pod \"2ae1daa1-61f5-4fae-a66b-e87d1895a8b5\" (UID: \"2ae1daa1-61f5-4fae-a66b-e87d1895a8b5\") " Nov 03 22:02:03.013197 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:03.013160 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae1daa1-61f5-4fae-a66b-e87d1895a8b5-kube-api-access-sjwqp" pod "2ae1daa1-61f5-4fae-a66b-e87d1895a8b5" (UID: "2ae1daa1-61f5-4fae-a66b-e87d1895a8b5"). InnerVolumeSpecName "kube-api-access-sjwqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:02:03.112484 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:03.112444 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjwqp\" (UniqueName: \"kubernetes.io/projected/2ae1daa1-61f5-4fae-a66b-e87d1895a8b5-kube-api-access-sjwqp\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:02:03.309532 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:03.309433 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7478-ce7155"] Nov 03 22:02:03.314661 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:03.314618 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7478-ce7155"] Nov 03 22:02:03.748925 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:03.744052 4963 generic.go:356] "Generic (PLEG): container finished" podID="2ae1daa1-61f5-4fae-a66b-e87d1895a8b5" containerID="832885e003de38ee5f79e08ea2e8fab4163907f48a14ed1137af092c9b2f80b9" exitCode=137 Nov 03 22:02:03.748925 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:03.744105 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-20c61c" Nov 03 22:02:03.748925 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:03.744142 4963 scope.go:122] "RemoveContainer" containerID="832885e003de38ee5f79e08ea2e8fab4163907f48a14ed1137af092c9b2f80b9" Nov 03 22:02:03.748925 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:03.744405 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7478-ce7155" podUID="f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42" containerName="test-curl-7478-ce7155" containerID="cri-o://49e414a4a34020bb723032488dcba91060297c448fe3257f577ede791d13a9be" gracePeriod=2 Nov 03 22:02:03.763155 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:03.763121 4963 scope.go:122] "RemoveContainer" containerID="832885e003de38ee5f79e08ea2e8fab4163907f48a14ed1137af092c9b2f80b9" Nov 03 22:02:03.763515 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:02:03.763488 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"832885e003de38ee5f79e08ea2e8fab4163907f48a14ed1137af092c9b2f80b9\": container with ID starting with 832885e003de38ee5f79e08ea2e8fab4163907f48a14ed1137af092c9b2f80b9 not found: ID does not exist" containerID="832885e003de38ee5f79e08ea2e8fab4163907f48a14ed1137af092c9b2f80b9" Nov 03 22:02:03.763593 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:03.763531 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832885e003de38ee5f79e08ea2e8fab4163907f48a14ed1137af092c9b2f80b9"} err="failed to get container status \"832885e003de38ee5f79e08ea2e8fab4163907f48a14ed1137af092c9b2f80b9\": rpc error: code = NotFound desc = could not find container \"832885e003de38ee5f79e08ea2e8fab4163907f48a14ed1137af092c9b2f80b9\": container with ID starting with 832885e003de38ee5f79e08ea2e8fab4163907f48a14ed1137af092c9b2f80b9 not found: ID does not exist" Nov 03 22:02:03.793209 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:02:03.793163 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7478-ce7155\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42" pod="opendatahub/test-curl-7478-ce7155" Nov 03 22:02:03.841811 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:02:03.841763 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7478-20c61c\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="2ae1daa1-61f5-4fae-a66b-e87d1895a8b5" pod="opendatahub/test-curl-7478-20c61c" Nov 03 22:02:03.890377 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:02:03.890325 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7478-ce7155\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42" pod="opendatahub/test-curl-7478-ce7155" Nov 03 22:02:04.861750 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:04.861714 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6"] Nov 03 22:02:04.863123 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:04.863091 4963 cpu_manager.go:414] "containerMap: removing container" podUID="f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42" containerName="test-curl-7478-ce7155" Nov 03 22:02:04.863311 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:04.863284 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42" containerName="test-curl-7478-ce7155" Nov 03 22:02:04.863527 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:04.863501 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42" containerName="test-curl-7478-ce7155" Nov 03 22:02:04.868669 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:04.868644 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:04.917581 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:02:04.917539 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7478-20c61c\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="2ae1daa1-61f5-4fae-a66b-e87d1895a8b5" pod="opendatahub/test-curl-7478-20c61c" Nov 03 22:02:04.921348 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:04.921314 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6"] Nov 03 22:02:04.940450 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:04.940411 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-tls-certs\") pod \"sub-test-model-c70fca-kserve-5559c94b5-wt7m6\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:04.940659 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:04.940559 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-tmp-dir\") pod \"sub-test-model-c70fca-kserve-5559c94b5-wt7m6\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:04.940659 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:04.940622 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjkpz\" (UniqueName: \"kubernetes.io/projected/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-kube-api-access-mjkpz\") pod \"sub-test-model-c70fca-kserve-5559c94b5-wt7m6\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:04.940773 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:04.940662 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-home\") pod \"sub-test-model-c70fca-kserve-5559c94b5-wt7m6\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:04.940773 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:04.940757 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-model-cache\") pod \"sub-test-model-c70fca-kserve-5559c94b5-wt7m6\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:04.940875 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:04.940822 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-dshm\") pod \"sub-test-model-c70fca-kserve-5559c94b5-wt7m6\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:04.965036 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:04.964998 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-c70fca\"/\"kube-root-ca.crt\"" Nov 03 22:02:04.965214 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:04.965152 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-c70fca\"/\"default-dockercfg-npcrx\"" Nov 03 22:02:04.965214 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:04.965152 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-c70fca\"/\"openshift-service-ca.crt\"" Nov 03 22:02:04.965334 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:04.965214 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-c70fca\"/\"sub-test-model-c70fca-kserve-self-signed-certs\"" Nov 03 22:02:04.966023 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:02:04.965998 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7478-ce7155\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42" pod="opendatahub/test-curl-7478-ce7155" Nov 03 22:02:04.967725 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:04.967695 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:04.970014 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:04.969990 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae1daa1-61f5-4fae-a66b-e87d1895a8b5" path="/var/lib/kubelet/pods/2ae1daa1-61f5-4fae-a66b-e87d1895a8b5/volumes" Nov 03 22:02:05.041800 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:05.041761 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-model-cache\") pod \"sub-test-model-c70fca-kserve-5559c94b5-wt7m6\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:05.042013 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:05.041837 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-dshm\") pod \"sub-test-model-c70fca-kserve-5559c94b5-wt7m6\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:05.042099 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:05.042037 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-tls-certs\") pod \"sub-test-model-c70fca-kserve-5559c94b5-wt7m6\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:05.042206 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:05.042181 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-tmp-dir\") pod \"sub-test-model-c70fca-kserve-5559c94b5-wt7m6\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:05.042278 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:05.042255 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjkpz\" (UniqueName: \"kubernetes.io/projected/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-kube-api-access-mjkpz\") pod \"sub-test-model-c70fca-kserve-5559c94b5-wt7m6\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:05.042336 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:05.042186 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-model-cache\") pod \"sub-test-model-c70fca-kserve-5559c94b5-wt7m6\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:05.042395 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:05.042317 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-home\") pod \"sub-test-model-c70fca-kserve-5559c94b5-wt7m6\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:05.042532 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:05.042504 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-tmp-dir\") pod \"sub-test-model-c70fca-kserve-5559c94b5-wt7m6\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:05.042675 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:05.042607 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-home\") pod \"sub-test-model-c70fca-kserve-5559c94b5-wt7m6\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:05.044427 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:05.044402 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-dshm\") pod \"sub-test-model-c70fca-kserve-5559c94b5-wt7m6\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:05.044608 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:05.044591 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-tls-certs\") pod \"sub-test-model-c70fca-kserve-5559c94b5-wt7m6\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:05.101098 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:05.101064 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjkpz\" (UniqueName: \"kubernetes.io/projected/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-kube-api-access-mjkpz\") pod \"sub-test-model-c70fca-kserve-5559c94b5-wt7m6\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:05.182439 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:05.182342 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:05.571026 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:05.570988 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6"] Nov 03 22:02:05.576533 ip-10-0-130-134 kubenswrapper[4963]: W1103 22:02:05.576498 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb3fc56a_e7a0_445a_b206_f903fd3d12d9.slice/crio-19cb79bd2a8d44e1d84f36c3b0906776503f2f684e6e3264aad035416a173f06 WatchSource:0}: Error finding container 19cb79bd2a8d44e1d84f36c3b0906776503f2f684e6e3264aad035416a173f06: Status 404 returned error can't find the container with id 19cb79bd2a8d44e1d84f36c3b0906776503f2f684e6e3264aad035416a173f06 Nov 03 22:02:05.760946 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:05.760861 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" event={"ID":"cb3fc56a-e7a0-445a-b206-f903fd3d12d9","Type":"ContainerStarted","Data":"daf068cae90b11a3dc4a2baac58dafe535286392b312bd8084329dacf8a34078"} Nov 03 22:02:05.760946 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:05.760922 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" event={"ID":"cb3fc56a-e7a0-445a-b206-f903fd3d12d9","Type":"ContainerStarted","Data":"19cb79bd2a8d44e1d84f36c3b0906776503f2f684e6e3264aad035416a173f06"} Nov 03 22:02:05.873824 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:05.873720 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" podStartSLOduration=1.873706101 podStartE2EDuration="1.873706101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:05.871965077 +0000 UTC m=+4475.032037351" watchObservedRunningTime="2026-11-03 22:02:05.873706101 +0000 UTC m=+4475.033778372" Nov 03 22:02:05.970282 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:05.970250 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-ce7155" Nov 03 22:02:06.054106 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:06.054059 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42-kube-api-access-l8qsl\" (UniqueName: \"kubernetes.io/projected/f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42-kube-api-access-l8qsl\") pod \"f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42\" (UID: \"f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42\") " Nov 03 22:02:06.056442 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:06.056416 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42-kube-api-access-l8qsl" pod "f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42" (UID: "f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42"). InnerVolumeSpecName "kube-api-access-l8qsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:02:06.155674 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:06.155593 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8qsl\" (UniqueName: \"kubernetes.io/projected/f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42-kube-api-access-l8qsl\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:02:06.183097 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:06.183047 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:06.768243 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:06.768212 4963 generic.go:356] "Generic (PLEG): container finished" podID="f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42" containerID="49e414a4a34020bb723032488dcba91060297c448fe3257f577ede791d13a9be" exitCode=137 Nov 03 22:02:06.768450 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:06.768255 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-ce7155" Nov 03 22:02:06.768450 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:06.768313 4963 scope.go:122] "RemoveContainer" containerID="49e414a4a34020bb723032488dcba91060297c448fe3257f577ede791d13a9be" Nov 03 22:02:06.778073 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:06.778058 4963 scope.go:122] "RemoveContainer" containerID="49e414a4a34020bb723032488dcba91060297c448fe3257f577ede791d13a9be" Nov 03 22:02:06.778311 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:02:06.778293 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49e414a4a34020bb723032488dcba91060297c448fe3257f577ede791d13a9be\": container with ID starting with 49e414a4a34020bb723032488dcba91060297c448fe3257f577ede791d13a9be not found: ID does not exist" containerID="49e414a4a34020bb723032488dcba91060297c448fe3257f577ede791d13a9be" Nov 03 22:02:06.778351 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:06.778320 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49e414a4a34020bb723032488dcba91060297c448fe3257f577ede791d13a9be"} err="failed to get container status \"49e414a4a34020bb723032488dcba91060297c448fe3257f577ede791d13a9be\": rpc error: code = NotFound desc = could not find container \"49e414a4a34020bb723032488dcba91060297c448fe3257f577ede791d13a9be\": container with ID starting with 49e414a4a34020bb723032488dcba91060297c448fe3257f577ede791d13a9be not found: ID does not exist" Nov 03 22:02:06.825738 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:02:06.825699 4963 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7478-ce7155\" is forbidden: User \"system:node:ip-10-0-130-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.us-west-2.compute.internal' and this object" podUID="f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42" pod="opendatahub/test-curl-7478-ce7155" Nov 03 22:02:06.967221 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:06.967189 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42" path="/var/lib/kubelet/pods/f1cb515f-e056-4ccb-8e79-a2ab6dbb5b42/volumes" Nov 03 22:02:07.050541 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:07.050464 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:09.907635 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:09.907602 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:12.527698 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:02:12.527663 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:02:14.511509 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:14.511469 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:15.182476 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:15.182444 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:15.195291 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:15.195264 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:15.836174 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:15.836139 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:02:19.020651 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:19.020598 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:23.614854 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:23.614787 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:28.159979 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:28.159948 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:31.588336 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:31.588303 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 22:02:31.610573 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:31.610546 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 22:02:32.728596 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:32.728552 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:37.317255 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:37.317217 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:41.912453 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:41.912420 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:42.528802 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:02:42.528760 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:02:46.456121 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:46.456089 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:46.634435 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:46.634388 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:49.525601 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:49.525560 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:51.059246 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.059214 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:51.134095 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.134052 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6"] Nov 03 22:02:51.138273 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.138251 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.209346 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.209309 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-tmp-dir\") pod \"sub-test-model-a1c634-kserve-6c5577c57-nhsd6\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.209552 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.209451 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr4vl\" (UniqueName: \"kubernetes.io/projected/d6b006c9-5373-4b54-b9a3-ae0ea5425005-kube-api-access-gr4vl\") pod \"sub-test-model-a1c634-kserve-6c5577c57-nhsd6\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.209645 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.209633 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-model-cache\") pod \"sub-test-model-a1c634-kserve-6c5577c57-nhsd6\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.209709 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.209664 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b006c9-5373-4b54-b9a3-ae0ea5425005-tls-certs\") pod \"sub-test-model-a1c634-kserve-6c5577c57-nhsd6\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.209769 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.209704 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-home\") pod \"sub-test-model-a1c634-kserve-6c5577c57-nhsd6\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.209847 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.209826 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-dshm\") pod \"sub-test-model-a1c634-kserve-6c5577c57-nhsd6\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.233433 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.233402 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-a1c634\"/\"kube-root-ca.crt\"" Nov 03 22:02:51.236679 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.236651 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-a1c634\"/\"openshift-service-ca.crt\"" Nov 03 22:02:51.237033 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.236972 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-a1c634\"/\"sub-test-model-a1c634-kserve-self-signed-certs\"" Nov 03 22:02:51.237140 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.237029 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-a1c634\"/\"default-dockercfg-zhztf\"" Nov 03 22:02:51.242150 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.242125 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6"] Nov 03 22:02:51.263020 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.262976 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:02:51.310879 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.310785 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b006c9-5373-4b54-b9a3-ae0ea5425005-tls-certs\") pod \"sub-test-model-a1c634-kserve-6c5577c57-nhsd6\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.310879 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.310838 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-home\") pod \"sub-test-model-a1c634-kserve-6c5577c57-nhsd6\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.311104 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.310885 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-dshm\") pod \"sub-test-model-a1c634-kserve-6c5577c57-nhsd6\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.311104 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.311047 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-tmp-dir\") pod \"sub-test-model-a1c634-kserve-6c5577c57-nhsd6\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.311177 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.311159 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr4vl\" (UniqueName: \"kubernetes.io/projected/d6b006c9-5373-4b54-b9a3-ae0ea5425005-kube-api-access-gr4vl\") pod \"sub-test-model-a1c634-kserve-6c5577c57-nhsd6\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.311345 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.311324 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-model-cache\") pod \"sub-test-model-a1c634-kserve-6c5577c57-nhsd6\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.311440 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.311416 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-home\") pod \"sub-test-model-a1c634-kserve-6c5577c57-nhsd6\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.311493 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.311466 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-tmp-dir\") pod \"sub-test-model-a1c634-kserve-6c5577c57-nhsd6\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.311606 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.311589 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-model-cache\") pod \"sub-test-model-a1c634-kserve-6c5577c57-nhsd6\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.313304 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.313282 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-dshm\") pod \"sub-test-model-a1c634-kserve-6c5577c57-nhsd6\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.313781 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.313757 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b006c9-5373-4b54-b9a3-ae0ea5425005-tls-certs\") pod \"sub-test-model-a1c634-kserve-6c5577c57-nhsd6\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.370614 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.370575 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr4vl\" (UniqueName: \"kubernetes.io/projected/d6b006c9-5373-4b54-b9a3-ae0ea5425005-kube-api-access-gr4vl\") pod \"sub-test-model-a1c634-kserve-6c5577c57-nhsd6\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.450835 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.450799 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:51.835403 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:51.835367 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6"] Nov 03 22:02:51.840147 ip-10-0-130-134 kubenswrapper[4963]: W1103 22:02:51.840105 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6b006c9_5373_4b54_b9a3_ae0ea5425005.slice/crio-364ac6ac0ee0d7affbdd8351fef078c07ec1adc08d3809b42993647b4a7bf420 WatchSource:0}: Error finding container 364ac6ac0ee0d7affbdd8351fef078c07ec1adc08d3809b42993647b4a7bf420: Status 404 returned error can't find the container with id 364ac6ac0ee0d7affbdd8351fef078c07ec1adc08d3809b42993647b4a7bf420 Nov 03 22:02:52.088015 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:52.087869 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" event={"ID":"d6b006c9-5373-4b54-b9a3-ae0ea5425005","Type":"ContainerStarted","Data":"63ea8eb21ad652ca18d9dff5caef9d83b34ceef0700780c0e691413bdd9c5a91"} Nov 03 22:02:52.088015 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:52.087952 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" event={"ID":"d6b006c9-5373-4b54-b9a3-ae0ea5425005","Type":"ContainerStarted","Data":"364ac6ac0ee0d7affbdd8351fef078c07ec1adc08d3809b42993647b4a7bf420"} Nov 03 22:02:52.202001 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:52.201922 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" podStartSLOduration=1.201896418 podStartE2EDuration="1.201896418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:52.199227136 +0000 UTC m=+4521.359299409" watchObservedRunningTime="2026-11-03 22:02:52.201896418 +0000 UTC m=+4521.361968690" Nov 03 22:02:52.451413 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:52.451381 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:02:55.579231 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:02:55.579199 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:00.064043 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:00.064009 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:01.451304 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:01.451263 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:03:01.464223 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:01.464192 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:03:02.170962 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:02.170932 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:03:04.537095 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:04.537048 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:06.612071 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:06.612023 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:09.113087 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:09.113051 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:12.532294 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:03:12.532209 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:03:13.227399 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:13.227360 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:13.756851 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:13.756813 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:14.255285 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:14.255255 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:14.495866 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:14.495836 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:18.510335 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:18.510291 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:19.137972 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:19.137937 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:20.568941 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:20.568888 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:22.166274 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:22.166235 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:22.566753 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:22.566712 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:23.165343 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:23.165309 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:26.782716 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:26.782681 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:27.161650 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:27.161566 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:27.355352 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:27.355318 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:27.812343 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:27.812311 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:32.521408 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:32.521369 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:32.704004 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:32.703882 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:37.115453 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:37.115420 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:39.241377 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:39.241338 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9"] Nov 03 22:03:39.241862 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:39.241586 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" podUID="98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" containerName="istio-proxy" containerID="cri-o://9136ae17435fb96c295861070551ecec36b61b2340cdc1e5289ac35928deb1a9" gracePeriod=30 Nov 03 22:03:41.645217 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:41.645184 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:42.529320 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:03:42.529275 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:03:44.210268 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.210231 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:03:44.210675 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.210306 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" podUID="98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:03:44.439478 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.439434 4963 generic.go:356] "Generic (PLEG): container finished" podID="98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" containerID="9136ae17435fb96c295861070551ecec36b61b2340cdc1e5289ac35928deb1a9" exitCode=0 Nov 03 22:03:44.439665 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.439594 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" event={"ID":"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd","Type":"ContainerDied","Data":"9136ae17435fb96c295861070551ecec36b61b2340cdc1e5289ac35928deb1a9"} Nov 03 22:03:44.510787 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.510758 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 22:03:44.570890 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.570852 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-workload-certs\") pod \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " Nov 03 22:03:44.571095 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.571035 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-credential-socket\") pod \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " Nov 03 22:03:44.571150 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.571106 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-data\") pod \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " Nov 03 22:03:44.571199 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.571172 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-envoy\") pod \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " Nov 03 22:03:44.571199 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.571179 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-workload-certs" pod "98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" (UID: "98ade2d0-82c2-4a55-9bfa-d6f5916c81fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:44.571317 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.571225 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-token\" (UniqueName: \"kubernetes.io/projected/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-token\") pod \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " Nov 03 22:03:44.571317 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.571265 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istiod-ca-cert\") pod \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " Nov 03 22:03:44.571433 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.571364 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-credential-socket" pod "98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" (UID: "98ade2d0-82c2-4a55-9bfa-d6f5916c81fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:44.571433 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.571412 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-data" pod "98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" (UID: "98ade2d0-82c2-4a55-9bfa-d6f5916c81fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:44.571433 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.571421 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-workload-socket\") pod \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " Nov 03 22:03:44.571594 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.571464 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-podinfo\") pod \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " Nov 03 22:03:44.571594 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.571494 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-kube-api-access-gmgb6\" (UniqueName: \"kubernetes.io/projected/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-kube-api-access-gmgb6\") pod \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\" (UID: \"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd\") " Nov 03 22:03:44.571705 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.571633 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istiod-ca-cert" pod "98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" (UID: "98ade2d0-82c2-4a55-9bfa-d6f5916c81fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:03:44.571705 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.571674 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-workload-socket" pod "98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" (UID: "98ade2d0-82c2-4a55-9bfa-d6f5916c81fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:44.572094 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.572071 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:44.572224 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.572100 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:44.572224 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.572115 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:44.572224 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.572129 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:44.572224 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.572143 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:44.573673 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.573644 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-token" pod "98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" (UID: "98ade2d0-82c2-4a55-9bfa-d6f5916c81fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:44.573673 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.573648 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-podinfo" pod "98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" (UID: "98ade2d0-82c2-4a55-9bfa-d6f5916c81fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:03:44.573945 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.573899 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-envoy" pod "98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" (UID: "98ade2d0-82c2-4a55-9bfa-d6f5916c81fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:44.574051 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.574037 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-kube-api-access-gmgb6" pod "98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" (UID: "98ade2d0-82c2-4a55-9bfa-d6f5916c81fd"). InnerVolumeSpecName "kube-api-access-gmgb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:44.673288 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.673251 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:44.673288 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.673281 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:44.673288 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.673290 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:44.673575 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:44.673302 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmgb6\" (UniqueName: \"kubernetes.io/projected/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd-kube-api-access-gmgb6\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:45.446884 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.446856 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" Nov 03 22:03:45.447345 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.446889 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9" event={"ID":"98ade2d0-82c2-4a55-9bfa-d6f5916c81fd","Type":"ContainerDied","Data":"1097962693b2e429e2181aa339a856c3ec4776a6e7ce227f07a32556c13fa5e9"} Nov 03 22:03:45.447345 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.446964 4963 scope.go:122] "RemoveContainer" containerID="9136ae17435fb96c295861070551ecec36b61b2340cdc1e5289ac35928deb1a9" Nov 03 22:03:45.470099 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.470067 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h"] Nov 03 22:03:45.470293 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.470259 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" podUID="4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a" containerName="maas-api" containerID="cri-o://1e40c4a14875311e74bbbb1719d0a5e828f7f5568e287fc2f9c405d922b74896" gracePeriod=30 Nov 03 22:03:45.602255 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.602225 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:45.737859 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.737824 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9"] Nov 03 22:03:45.743277 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.743247 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-vlgf9"] Nov 03 22:03:45.746490 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.746466 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:03:45.887412 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.887379 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-metrics-certs\") pod \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\" (UID: \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\") " Nov 03 22:03:45.887615 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.887431 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-kube-api-access-rf6s2\" (UniqueName: \"kubernetes.io/projected/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-kube-api-access-rf6s2\") pod \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\" (UID: \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\") " Nov 03 22:03:45.887615 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.887462 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-maas-api-tls\") pod \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\" (UID: \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\") " Nov 03 22:03:45.887615 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.887546 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-openshift-service-ca\") pod \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\" (UID: \"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a\") " Nov 03 22:03:45.888054 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.888022 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-openshift-service-ca" pod "4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a" (UID: "4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:03:45.888194 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.888135 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:45.889789 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.889750 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-metrics-certs" pod "4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a" (UID: "4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:03:45.890230 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.890205 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-kube-api-access-rf6s2" pod "4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a" (UID: "4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a"). InnerVolumeSpecName "kube-api-access-rf6s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:45.890347 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.890328 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-maas-api-tls" pod "4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a" (UID: "4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:03:45.989362 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.989265 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:45.989362 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.989306 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf6s2\" (UniqueName: \"kubernetes.io/projected/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-kube-api-access-rf6s2\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:45.989362 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:45.989323 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:46.455436 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:46.455402 4963 generic.go:356] "Generic (PLEG): container finished" podID="4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a" containerID="1e40c4a14875311e74bbbb1719d0a5e828f7f5568e287fc2f9c405d922b74896" exitCode=0 Nov 03 22:03:46.455875 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:46.455463 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" Nov 03 22:03:46.455875 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:46.455488 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" event={"ID":"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a","Type":"ContainerDied","Data":"1e40c4a14875311e74bbbb1719d0a5e828f7f5568e287fc2f9c405d922b74896"} Nov 03 22:03:46.455875 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:46.455560 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h" event={"ID":"4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a","Type":"ContainerDied","Data":"3a4ff940d995252f6ea0d554a818e9a385c7155efa89be47b3307e65288c3370"} Nov 03 22:03:46.455875 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:46.455582 4963 scope.go:122] "RemoveContainer" containerID="1e40c4a14875311e74bbbb1719d0a5e828f7f5568e287fc2f9c405d922b74896" Nov 03 22:03:46.468151 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:46.468126 4963 scope.go:122] "RemoveContainer" containerID="1e40c4a14875311e74bbbb1719d0a5e828f7f5568e287fc2f9c405d922b74896" Nov 03 22:03:46.468491 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:03:46.468463 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e40c4a14875311e74bbbb1719d0a5e828f7f5568e287fc2f9c405d922b74896\": container with ID starting with 1e40c4a14875311e74bbbb1719d0a5e828f7f5568e287fc2f9c405d922b74896 not found: ID does not exist" containerID="1e40c4a14875311e74bbbb1719d0a5e828f7f5568e287fc2f9c405d922b74896" Nov 03 22:03:46.468621 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:46.468501 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e40c4a14875311e74bbbb1719d0a5e828f7f5568e287fc2f9c405d922b74896"} err="failed to get container status \"1e40c4a14875311e74bbbb1719d0a5e828f7f5568e287fc2f9c405d922b74896\": rpc error: code = NotFound desc = could not find container \"1e40c4a14875311e74bbbb1719d0a5e828f7f5568e287fc2f9c405d922b74896\": container with ID starting with 1e40c4a14875311e74bbbb1719d0a5e828f7f5568e287fc2f9c405d922b74896 not found: ID does not exist" Nov 03 22:03:46.612276 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:46.612243 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h"] Nov 03 22:03:46.616102 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:46.616068 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85db4aad-7586748c79-h9l4h"] Nov 03 22:03:46.808892 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:46.808813 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:46.967826 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:46.967791 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a" path="/var/lib/kubelet/pods/4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a/volumes" Nov 03 22:03:46.968359 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:46.968342 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" path="/var/lib/kubelet/pods/98ade2d0-82c2-4a55-9bfa-d6f5916c81fd/volumes" Nov 03 22:03:47.227861 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:47.227826 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:51.725803 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:51.725762 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:52.591881 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:52.591848 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:52.778524 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:52.778485 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:54.092182 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:54.092149 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:54.363727 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:54.363614 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc"] Nov 03 22:03:54.363934 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:54.363833 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" podUID="70e4f9f6-f644-40b2-93fb-0acb72d86443" containerName="istio-proxy" containerID="cri-o://a51f11f7e0c775a7b707c28ab576b0744ef3d221aec9b9c3afacadefd83e90ff" gracePeriod=30 Nov 03 22:03:54.738669 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:54.738628 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:55.230651 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:55.230620 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:55.866915 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:55.866863 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:57.629247 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:57.629200 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:58.253818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:58.253783 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:58.504163 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:58.504062 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:03:59.540164 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.539944 4963 generic.go:356] "Generic (PLEG): container finished" podID="70e4f9f6-f644-40b2-93fb-0acb72d86443" containerID="a51f11f7e0c775a7b707c28ab576b0744ef3d221aec9b9c3afacadefd83e90ff" exitCode=0 Nov 03 22:03:59.540164 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.540043 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" event={"ID":"70e4f9f6-f644-40b2-93fb-0acb72d86443","Type":"ContainerDied","Data":"a51f11f7e0c775a7b707c28ab576b0744ef3d221aec9b9c3afacadefd83e90ff"} Nov 03 22:03:59.646198 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.646131 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 22:03:59.658000 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.657969 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-envoy\") pod \"70e4f9f6-f644-40b2-93fb-0acb72d86443\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " Nov 03 22:03:59.658170 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.658026 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e4f9f6-f644-40b2-93fb-0acb72d86443-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70e4f9f6-f644-40b2-93fb-0acb72d86443-istiod-ca-cert\") pod \"70e4f9f6-f644-40b2-93fb-0acb72d86443\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " Nov 03 22:03:59.658170 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.658058 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-podinfo\") pod \"70e4f9f6-f644-40b2-93fb-0acb72d86443\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " Nov 03 22:03:59.658170 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.658093 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-credential-socket\") pod \"70e4f9f6-f644-40b2-93fb-0acb72d86443\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " Nov 03 22:03:59.658170 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.658164 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e4f9f6-f644-40b2-93fb-0acb72d86443-kube-api-access-fstj6\" (UniqueName: \"kubernetes.io/projected/70e4f9f6-f644-40b2-93fb-0acb72d86443-kube-api-access-fstj6\") pod \"70e4f9f6-f644-40b2-93fb-0acb72d86443\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " Nov 03 22:03:59.658403 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.658191 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-workload-certs\") pod \"70e4f9f6-f644-40b2-93fb-0acb72d86443\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " Nov 03 22:03:59.658403 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.658232 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-token\" (UniqueName: \"kubernetes.io/projected/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-token\") pod \"70e4f9f6-f644-40b2-93fb-0acb72d86443\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " Nov 03 22:03:59.658403 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.658269 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-workload-socket\") pod \"70e4f9f6-f644-40b2-93fb-0acb72d86443\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " Nov 03 22:03:59.658403 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.658299 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-data\") pod \"70e4f9f6-f644-40b2-93fb-0acb72d86443\" (UID: \"70e4f9f6-f644-40b2-93fb-0acb72d86443\") " Nov 03 22:03:59.658703 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.658457 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-credential-socket" pod "70e4f9f6-f644-40b2-93fb-0acb72d86443" (UID: "70e4f9f6-f644-40b2-93fb-0acb72d86443"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:59.658703 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.658476 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e4f9f6-f644-40b2-93fb-0acb72d86443-istiod-ca-cert" pod "70e4f9f6-f644-40b2-93fb-0acb72d86443" (UID: "70e4f9f6-f644-40b2-93fb-0acb72d86443"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:03:59.658703 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.658686 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-data" pod "70e4f9f6-f644-40b2-93fb-0acb72d86443" (UID: "70e4f9f6-f644-40b2-93fb-0acb72d86443"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:59.658876 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.658703 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-workload-certs" pod "70e4f9f6-f644-40b2-93fb-0acb72d86443" (UID: "70e4f9f6-f644-40b2-93fb-0acb72d86443"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:59.658876 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.658766 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-workload-socket" pod "70e4f9f6-f644-40b2-93fb-0acb72d86443" (UID: "70e4f9f6-f644-40b2-93fb-0acb72d86443"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:59.659039 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.658973 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70e4f9f6-f644-40b2-93fb-0acb72d86443-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:59.659039 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.658997 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:59.659039 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.659008 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:59.659039 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.659016 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:59.659039 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.659025 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:59.660448 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.660425 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-podinfo" pod "70e4f9f6-f644-40b2-93fb-0acb72d86443" (UID: "70e4f9f6-f644-40b2-93fb-0acb72d86443"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:03:59.660674 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.660646 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e4f9f6-f644-40b2-93fb-0acb72d86443-kube-api-access-fstj6" pod "70e4f9f6-f644-40b2-93fb-0acb72d86443" (UID: "70e4f9f6-f644-40b2-93fb-0acb72d86443"). InnerVolumeSpecName "kube-api-access-fstj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:59.660773 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.660701 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-envoy" pod "70e4f9f6-f644-40b2-93fb-0acb72d86443" (UID: "70e4f9f6-f644-40b2-93fb-0acb72d86443"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:59.660773 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.660725 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-token" pod "70e4f9f6-f644-40b2-93fb-0acb72d86443" (UID: "70e4f9f6-f644-40b2-93fb-0acb72d86443"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:59.760490 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.760444 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:59.760490 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.760482 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fstj6\" (UniqueName: \"kubernetes.io/projected/70e4f9f6-f644-40b2-93fb-0acb72d86443-kube-api-access-fstj6\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:59.760490 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.760494 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:59.760490 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.760502 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70e4f9f6-f644-40b2-93fb-0acb72d86443-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:03:59.780349 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:03:59.780301 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:00.312178 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:00.312142 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br"] Nov 03 22:04:00.380766 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:00.380727 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:00.548144 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:00.548114 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" Nov 03 22:04:00.548144 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:00.548136 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" event={"ID":"70e4f9f6-f644-40b2-93fb-0acb72d86443","Type":"ContainerDied","Data":"237ad02c8857f956664667a5d15bf07e65a52ff6b274d1800bf1a113e70604be"} Nov 03 22:04:00.548659 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:00.548197 4963 scope.go:122] "RemoveContainer" containerID="a51f11f7e0c775a7b707c28ab576b0744ef3d221aec9b9c3afacadefd83e90ff" Nov 03 22:04:00.561974 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:00.561943 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:00.638513 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:00.638406 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:04:00.638513 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:00.638488 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc" podUID="70e4f9f6-f644-40b2-93fb-0acb72d86443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:04:00.705064 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:00.705025 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc"] Nov 03 22:04:00.752235 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:00.752191 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9n7nc"] Nov 03 22:04:00.967834 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:00.967799 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70e4f9f6-f644-40b2-93fb-0acb72d86443" path="/var/lib/kubelet/pods/70e4f9f6-f644-40b2-93fb-0acb72d86443/volumes" Nov 03 22:04:01.102390 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:01.102352 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:02.409980 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:02.409925 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:06.958994 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:06.958954 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:07.175640 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:07.175613 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:07.712810 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:07.712778 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:11.737790 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:11.737752 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:12.529309 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:04:12.529265 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:04:12.827718 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:12.827629 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9"] Nov 03 22:04:12.937049 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:12.936988 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq"] Nov 03 22:04:13.262484 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:13.262449 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:13.450314 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:13.450284 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:14.339890 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:14.339857 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:15.334528 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.334490 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" podUID="a3f5464f-5d94-499e-9fee-7553aa1e78a4" containerName="main" containerID="cri-o://8656024f417dfca40269a9d4d37d6e355bf39e0745d28858891b91fcb5e0821a" gracePeriod=45 Nov 03 22:04:15.604976 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.604871 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 22:04:15.640623 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.640586 4963 generic.go:356] "Generic (PLEG): container finished" podID="a3f5464f-5d94-499e-9fee-7553aa1e78a4" containerID="8656024f417dfca40269a9d4d37d6e355bf39e0745d28858891b91fcb5e0821a" exitCode=0 Nov 03 22:04:15.640869 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.640634 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" Nov 03 22:04:15.640869 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.640672 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" event={"ID":"a3f5464f-5d94-499e-9fee-7553aa1e78a4","Type":"ContainerDied","Data":"8656024f417dfca40269a9d4d37d6e355bf39e0745d28858891b91fcb5e0821a"} Nov 03 22:04:15.640869 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.640715 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br" event={"ID":"a3f5464f-5d94-499e-9fee-7553aa1e78a4","Type":"ContainerDied","Data":"3af4120146630ca8cff9b1ec76f1d052a1b88e7327cee0f27942e6948d8e79d0"} Nov 03 22:04:15.640869 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.640733 4963 scope.go:122] "RemoveContainer" containerID="8656024f417dfca40269a9d4d37d6e355bf39e0745d28858891b91fcb5e0821a" Nov 03 22:04:15.651690 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.651666 4963 scope.go:122] "RemoveContainer" containerID="8656024f417dfca40269a9d4d37d6e355bf39e0745d28858891b91fcb5e0821a" Nov 03 22:04:15.652063 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:04:15.652036 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8656024f417dfca40269a9d4d37d6e355bf39e0745d28858891b91fcb5e0821a\": container with ID starting with 8656024f417dfca40269a9d4d37d6e355bf39e0745d28858891b91fcb5e0821a not found: ID does not exist" containerID="8656024f417dfca40269a9d4d37d6e355bf39e0745d28858891b91fcb5e0821a" Nov 03 22:04:15.652215 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.652067 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8656024f417dfca40269a9d4d37d6e355bf39e0745d28858891b91fcb5e0821a"} err="failed to get container status \"8656024f417dfca40269a9d4d37d6e355bf39e0745d28858891b91fcb5e0821a\": rpc error: code = NotFound desc = could not find container \"8656024f417dfca40269a9d4d37d6e355bf39e0745d28858891b91fcb5e0821a\": container with ID starting with 8656024f417dfca40269a9d4d37d6e355bf39e0745d28858891b91fcb5e0821a not found: ID does not exist" Nov 03 22:04:15.745769 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.745732 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3f5464f-5d94-499e-9fee-7553aa1e78a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f5464f-5d94-499e-9fee-7553aa1e78a4-tls-certs\") pod \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " Nov 03 22:04:15.746018 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.745802 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-model-cache\") pod \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " Nov 03 22:04:15.746018 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.745830 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-dshm\") pod \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " Nov 03 22:04:15.746018 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.745913 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-tmp-dir\") pod \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " Nov 03 22:04:15.746018 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.745997 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f5464f-5d94-499e-9fee-7553aa1e78a4-kube-api-access-sddt2\" (UniqueName: \"kubernetes.io/projected/a3f5464f-5d94-499e-9fee-7553aa1e78a4-kube-api-access-sddt2\") pod \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " Nov 03 22:04:15.746270 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.746158 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-home\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-home\") pod \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\" (UID: \"a3f5464f-5d94-499e-9fee-7553aa1e78a4\") " Nov 03 22:04:15.746556 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.746502 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-tmp-dir" pod "a3f5464f-5d94-499e-9fee-7553aa1e78a4" (UID: "a3f5464f-5d94-499e-9fee-7553aa1e78a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:04:15.746756 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.746552 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-home" pod "a3f5464f-5d94-499e-9fee-7553aa1e78a4" (UID: "a3f5464f-5d94-499e-9fee-7553aa1e78a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:04:15.746880 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.746673 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-model-cache" pod "a3f5464f-5d94-499e-9fee-7553aa1e78a4" (UID: "a3f5464f-5d94-499e-9fee-7553aa1e78a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:04:15.748589 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.748562 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-dshm" pod "a3f5464f-5d94-499e-9fee-7553aa1e78a4" (UID: "a3f5464f-5d94-499e-9fee-7553aa1e78a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:04:15.749255 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.749228 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f5464f-5d94-499e-9fee-7553aa1e78a4-kube-api-access-sddt2" pod "a3f5464f-5d94-499e-9fee-7553aa1e78a4" (UID: "a3f5464f-5d94-499e-9fee-7553aa1e78a4"). InnerVolumeSpecName "kube-api-access-sddt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:04:15.749367 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.749294 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3f5464f-5d94-499e-9fee-7553aa1e78a4-tls-certs" pod "a3f5464f-5d94-499e-9fee-7553aa1e78a4" (UID: "a3f5464f-5d94-499e-9fee-7553aa1e78a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:04:15.847884 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.847847 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:15.847884 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.847887 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f5464f-5d94-499e-9fee-7553aa1e78a4-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:15.848107 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.847916 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:15.848107 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.847931 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:15.848107 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.847944 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f5464f-5d94-499e-9fee-7553aa1e78a4-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:15.848107 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:15.847958 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sddt2\" (UniqueName: \"kubernetes.io/projected/a3f5464f-5d94-499e-9fee-7553aa1e78a4-kube-api-access-sddt2\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:16.066660 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:16.066622 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br"] Nov 03 22:04:16.069170 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:16.069141 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/simulator-w3-320f60-kserve-6858fcc8bf-4b2br"] Nov 03 22:04:16.528331 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:16.528297 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:16.967811 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:16.967778 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3f5464f-5d94-499e-9fee-7553aa1e78a4" path="/var/lib/kubelet/pods/a3f5464f-5d94-499e-9fee-7553aa1e78a4/volumes" Nov 03 22:04:17.828888 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:17.828815 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9" podUID="58309a66-dfdf-4793-bbc5-5c7b33ca111e" containerName="payload-processing" containerID="cri-o://9f5a1f12e08d4b55e677b1cf3391d4a38f0e438d8de8f3a51ad43bb4a0b7e449" gracePeriod=25 Nov 03 22:04:17.937681 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:17.937632 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq" podUID="9204dab2-ccbe-4405-b9c7-666f23309bbd" containerName="payload-pre-processing" containerID="cri-o://5b101cf0b10f79b1cb8d3d7fe1f774e56a2988ca2e1bad943c2632c3c8ba683b" gracePeriod=25 Nov 03 22:04:18.119684 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.119652 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9" Nov 03 22:04:18.223396 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.223364 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq" Nov 03 22:04:18.274792 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.274751 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58309a66-dfdf-4793-bbc5-5c7b33ca111e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58309a66-dfdf-4793-bbc5-5c7b33ca111e-plugins-config-volume\") pod \"58309a66-dfdf-4793-bbc5-5c7b33ca111e\" (UID: \"58309a66-dfdf-4793-bbc5-5c7b33ca111e\") " Nov 03 22:04:18.275010 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.274983 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58309a66-dfdf-4793-bbc5-5c7b33ca111e-kube-api-access-j5n6p\" (UniqueName: \"kubernetes.io/projected/58309a66-dfdf-4793-bbc5-5c7b33ca111e-kube-api-access-j5n6p\") pod \"58309a66-dfdf-4793-bbc5-5c7b33ca111e\" (UID: \"58309a66-dfdf-4793-bbc5-5c7b33ca111e\") " Nov 03 22:04:18.275295 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.275264 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58309a66-dfdf-4793-bbc5-5c7b33ca111e-plugins-config-volume" pod "58309a66-dfdf-4793-bbc5-5c7b33ca111e" (UID: "58309a66-dfdf-4793-bbc5-5c7b33ca111e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:04:18.277274 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.277238 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58309a66-dfdf-4793-bbc5-5c7b33ca111e-kube-api-access-j5n6p" pod "58309a66-dfdf-4793-bbc5-5c7b33ca111e" (UID: "58309a66-dfdf-4793-bbc5-5c7b33ca111e"). InnerVolumeSpecName "kube-api-access-j5n6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:04:18.376622 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.376523 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9204dab2-ccbe-4405-b9c7-666f23309bbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9204dab2-ccbe-4405-b9c7-666f23309bbd-plugins-config-volume\") pod \"9204dab2-ccbe-4405-b9c7-666f23309bbd\" (UID: \"9204dab2-ccbe-4405-b9c7-666f23309bbd\") " Nov 03 22:04:18.376622 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.376576 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9204dab2-ccbe-4405-b9c7-666f23309bbd-kube-api-access-brflb\" (UniqueName: \"kubernetes.io/projected/9204dab2-ccbe-4405-b9c7-666f23309bbd-kube-api-access-brflb\") pod \"9204dab2-ccbe-4405-b9c7-666f23309bbd\" (UID: \"9204dab2-ccbe-4405-b9c7-666f23309bbd\") " Nov 03 22:04:18.376977 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.376951 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9204dab2-ccbe-4405-b9c7-666f23309bbd-plugins-config-volume" pod "9204dab2-ccbe-4405-b9c7-666f23309bbd" (UID: "9204dab2-ccbe-4405-b9c7-666f23309bbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:04:18.377080 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.377033 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5n6p\" (UniqueName: \"kubernetes.io/projected/58309a66-dfdf-4793-bbc5-5c7b33ca111e-kube-api-access-j5n6p\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:18.377080 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.377050 4963 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58309a66-dfdf-4793-bbc5-5c7b33ca111e-plugins-config-volume\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:18.379068 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.378898 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9204dab2-ccbe-4405-b9c7-666f23309bbd-kube-api-access-brflb" pod "9204dab2-ccbe-4405-b9c7-666f23309bbd" (UID: "9204dab2-ccbe-4405-b9c7-666f23309bbd"). InnerVolumeSpecName "kube-api-access-brflb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:04:18.478499 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.478463 4963 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9204dab2-ccbe-4405-b9c7-666f23309bbd-plugins-config-volume\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:18.478499 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.478494 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brflb\" (UniqueName: \"kubernetes.io/projected/9204dab2-ccbe-4405-b9c7-666f23309bbd-kube-api-access-brflb\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:04:18.661647 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.661565 4963 generic.go:356] "Generic (PLEG): container finished" podID="9204dab2-ccbe-4405-b9c7-666f23309bbd" containerID="5b101cf0b10f79b1cb8d3d7fe1f774e56a2988ca2e1bad943c2632c3c8ba683b" exitCode=0 Nov 03 22:04:18.661647 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.661613 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq" Nov 03 22:04:18.661882 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.661648 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq" event={"ID":"9204dab2-ccbe-4405-b9c7-666f23309bbd","Type":"ContainerDied","Data":"5b101cf0b10f79b1cb8d3d7fe1f774e56a2988ca2e1bad943c2632c3c8ba683b"} Nov 03 22:04:18.661882 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.661704 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq" event={"ID":"9204dab2-ccbe-4405-b9c7-666f23309bbd","Type":"ContainerDied","Data":"9b7b77f5cd7d973fb88007c827a2c4a9f687c103fde18b61e41d3216e7593fea"} Nov 03 22:04:18.661882 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.661725 4963 scope.go:122] "RemoveContainer" containerID="5b101cf0b10f79b1cb8d3d7fe1f774e56a2988ca2e1bad943c2632c3c8ba683b" Nov 03 22:04:18.663197 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.663172 4963 generic.go:356] "Generic (PLEG): container finished" podID="58309a66-dfdf-4793-bbc5-5c7b33ca111e" containerID="9f5a1f12e08d4b55e677b1cf3391d4a38f0e438d8de8f3a51ad43bb4a0b7e449" exitCode=0 Nov 03 22:04:18.663338 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.663197 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9" event={"ID":"58309a66-dfdf-4793-bbc5-5c7b33ca111e","Type":"ContainerDied","Data":"9f5a1f12e08d4b55e677b1cf3391d4a38f0e438d8de8f3a51ad43bb4a0b7e449"} Nov 03 22:04:18.663338 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.663231 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9" Nov 03 22:04:18.663338 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.663245 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9" event={"ID":"58309a66-dfdf-4793-bbc5-5c7b33ca111e","Type":"ContainerDied","Data":"9ec0a8492615b1318cf3e1919daf7076f5cc09b4986ca8d58520027cc3a1b77f"} Nov 03 22:04:18.675519 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.675127 4963 scope.go:122] "RemoveContainer" containerID="5b101cf0b10f79b1cb8d3d7fe1f774e56a2988ca2e1bad943c2632c3c8ba683b" Nov 03 22:04:18.675593 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:04:18.675525 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b101cf0b10f79b1cb8d3d7fe1f774e56a2988ca2e1bad943c2632c3c8ba683b\": container with ID starting with 5b101cf0b10f79b1cb8d3d7fe1f774e56a2988ca2e1bad943c2632c3c8ba683b not found: ID does not exist" containerID="5b101cf0b10f79b1cb8d3d7fe1f774e56a2988ca2e1bad943c2632c3c8ba683b" Nov 03 22:04:18.675593 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.675559 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b101cf0b10f79b1cb8d3d7fe1f774e56a2988ca2e1bad943c2632c3c8ba683b"} err="failed to get container status \"5b101cf0b10f79b1cb8d3d7fe1f774e56a2988ca2e1bad943c2632c3c8ba683b\": rpc error: code = NotFound desc = could not find container \"5b101cf0b10f79b1cb8d3d7fe1f774e56a2988ca2e1bad943c2632c3c8ba683b\": container with ID starting with 5b101cf0b10f79b1cb8d3d7fe1f774e56a2988ca2e1bad943c2632c3c8ba683b not found: ID does not exist" Nov 03 22:04:18.675593 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.675583 4963 scope.go:122] "RemoveContainer" containerID="9f5a1f12e08d4b55e677b1cf3391d4a38f0e438d8de8f3a51ad43bb4a0b7e449" Nov 03 22:04:18.685514 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.685488 4963 scope.go:122] "RemoveContainer" containerID="9f5a1f12e08d4b55e677b1cf3391d4a38f0e438d8de8f3a51ad43bb4a0b7e449" Nov 03 22:04:18.685806 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:04:18.685783 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5a1f12e08d4b55e677b1cf3391d4a38f0e438d8de8f3a51ad43bb4a0b7e449\": container with ID starting with 9f5a1f12e08d4b55e677b1cf3391d4a38f0e438d8de8f3a51ad43bb4a0b7e449 not found: ID does not exist" containerID="9f5a1f12e08d4b55e677b1cf3391d4a38f0e438d8de8f3a51ad43bb4a0b7e449" Nov 03 22:04:18.685880 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.685814 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5a1f12e08d4b55e677b1cf3391d4a38f0e438d8de8f3a51ad43bb4a0b7e449"} err="failed to get container status \"9f5a1f12e08d4b55e677b1cf3391d4a38f0e438d8de8f3a51ad43bb4a0b7e449\": rpc error: code = NotFound desc = could not find container \"9f5a1f12e08d4b55e677b1cf3391d4a38f0e438d8de8f3a51ad43bb4a0b7e449\": container with ID starting with 9f5a1f12e08d4b55e677b1cf3391d4a38f0e438d8de8f3a51ad43bb4a0b7e449 not found: ID does not exist" Nov 03 22:04:18.817958 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.817897 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9"] Nov 03 22:04:18.864962 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.864926 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f0dc66-c84455dd4-zjqr9"] Nov 03 22:04:18.925561 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.925467 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq"] Nov 03 22:04:18.929654 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.929627 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0dc66-6ddbc589d7-rbdcq"] Nov 03 22:04:18.967063 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.967030 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58309a66-dfdf-4793-bbc5-5c7b33ca111e" path="/var/lib/kubelet/pods/58309a66-dfdf-4793-bbc5-5c7b33ca111e/volumes" Nov 03 22:04:18.967483 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:18.967468 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9204dab2-ccbe-4405-b9c7-666f23309bbd" path="/var/lib/kubelet/pods/9204dab2-ccbe-4405-b9c7-666f23309bbd/volumes" Nov 03 22:04:20.310436 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:20.310390 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:20.497337 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:20.497303 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:20.690655 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:20.690528 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:21.306832 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:21.306787 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:24.116894 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:24.116836 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:25.905898 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:25.905853 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:26.566603 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:26.566570 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:30.663748 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:30.663705 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:35.053789 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:35.053753 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:35.366312 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:35.366209 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:36.923699 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:36.923652 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:37.169129 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:37.169094 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:38.461104 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:38.461062 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:39.273238 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:39.273208 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:40.972958 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:40.972844 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:42.159060 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:42.159019 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:42.356242 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:42.356198 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:42.528345 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:04:42.528309 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:04:43.779113 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:43.779081 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:44.337713 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:44.337664 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:45.671208 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:45.671172 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:45.888065 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:45.888027 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:47.172719 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:47.172684 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:50.417621 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:50.417588 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:55.042126 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:55.042087 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:04:59.651178 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:04:59.651140 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:03.626345 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:03.626276 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:05.324780 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:05.324743 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:08.285839 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:08.285807 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:08.469571 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:08.469538 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:09.863789 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:09.863756 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:10.043314 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:10.043279 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:10.349110 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:10.349067 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:10.838032 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:10.837996 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:12.529032 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:05:12.528993 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:05:14.624277 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:14.624241 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:15.766723 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:15.766684 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9"] Nov 03 22:05:15.813969 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:15.813900 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55"] Nov 03 22:05:15.928224 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:15.928195 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:17.694532 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:17.694484 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:19.313136 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:19.313103 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:24.061386 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:24.061339 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:24.249076 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:24.249013 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:26.848572 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:26.848540 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:27.031070 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:27.030970 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:28.882918 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:28.882877 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:30.605851 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:30.605810 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:30.790925 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:30.790381 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" podUID="b48999ea-6a3b-427c-a22b-621a5cc997db" containerName="main" containerID="cri-o://9ad65fc1ac7e589fd7b5fa327835326ba809b6c7d0df78e390b21a087c37b34c" gracePeriod=45 Nov 03 22:05:30.841577 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:30.841082 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" podUID="cf9455eb-457d-4fec-aca7-f55202209d2c" containerName="main" containerID="cri-o://62952a2812750d957e5b0940b6de5bdaf5856b9cec924dd06ce80c9f88b18b4d" gracePeriod=45 Nov 03 22:05:31.151595 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.151505 4963 generic.go:356] "Generic (PLEG): container finished" podID="cf9455eb-457d-4fec-aca7-f55202209d2c" containerID="62952a2812750d957e5b0940b6de5bdaf5856b9cec924dd06ce80c9f88b18b4d" exitCode=0 Nov 03 22:05:31.152051 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.152023 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" event={"ID":"cf9455eb-457d-4fec-aca7-f55202209d2c","Type":"ContainerDied","Data":"62952a2812750d957e5b0940b6de5bdaf5856b9cec924dd06ce80c9f88b18b4d"} Nov 03 22:05:31.172423 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.172238 4963 generic.go:356] "Generic (PLEG): container finished" podID="b48999ea-6a3b-427c-a22b-621a5cc997db" containerID="9ad65fc1ac7e589fd7b5fa327835326ba809b6c7d0df78e390b21a087c37b34c" exitCode=0 Nov 03 22:05:31.172423 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.172368 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" event={"ID":"b48999ea-6a3b-427c-a22b-621a5cc997db","Type":"ContainerDied","Data":"9ad65fc1ac7e589fd7b5fa327835326ba809b6c7d0df78e390b21a087c37b34c"} Nov 03 22:05:31.456946 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.456882 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 22:05:31.564123 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.564085 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 22:05:31.652819 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.652715 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-tmp-dir\") pod \"cf9455eb-457d-4fec-aca7-f55202209d2c\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " Nov 03 22:05:31.652819 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.652796 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-model-cache\") pod \"cf9455eb-457d-4fec-aca7-f55202209d2c\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " Nov 03 22:05:31.653432 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.652839 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-home\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-home\") pod \"cf9455eb-457d-4fec-aca7-f55202209d2c\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " Nov 03 22:05:31.653432 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.653032 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-tmp-dir\") pod \"b48999ea-6a3b-427c-a22b-621a5cc997db\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " Nov 03 22:05:31.653432 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.653076 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf9455eb-457d-4fec-aca7-f55202209d2c-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf9455eb-457d-4fec-aca7-f55202209d2c-tls-certs\") pod \"cf9455eb-457d-4fec-aca7-f55202209d2c\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " Nov 03 22:05:31.653432 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.653108 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9455eb-457d-4fec-aca7-f55202209d2c-kube-api-access-gq9bg\" (UniqueName: \"kubernetes.io/projected/cf9455eb-457d-4fec-aca7-f55202209d2c-kube-api-access-gq9bg\") pod \"cf9455eb-457d-4fec-aca7-f55202209d2c\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " Nov 03 22:05:31.653432 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.653162 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-model-cache\") pod \"b48999ea-6a3b-427c-a22b-621a5cc997db\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " Nov 03 22:05:31.653432 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.653223 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b48999ea-6a3b-427c-a22b-621a5cc997db-tls-certs\" (UniqueName: \"kubernetes.io/secret/b48999ea-6a3b-427c-a22b-621a5cc997db-tls-certs\") pod \"b48999ea-6a3b-427c-a22b-621a5cc997db\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " Nov 03 22:05:31.653432 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.653247 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-dshm\") pod \"cf9455eb-457d-4fec-aca7-f55202209d2c\" (UID: \"cf9455eb-457d-4fec-aca7-f55202209d2c\") " Nov 03 22:05:31.653432 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.653279 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-dshm\") pod \"b48999ea-6a3b-427c-a22b-621a5cc997db\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " Nov 03 22:05:31.653432 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.653305 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48999ea-6a3b-427c-a22b-621a5cc997db-kube-api-access-h99jc\" (UniqueName: \"kubernetes.io/projected/b48999ea-6a3b-427c-a22b-621a5cc997db-kube-api-access-h99jc\") pod \"b48999ea-6a3b-427c-a22b-621a5cc997db\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " Nov 03 22:05:31.653432 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.653323 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-home\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-home\") pod \"b48999ea-6a3b-427c-a22b-621a5cc997db\" (UID: \"b48999ea-6a3b-427c-a22b-621a5cc997db\") " Nov 03 22:05:31.654014 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.653802 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-home" pod "b48999ea-6a3b-427c-a22b-621a5cc997db" (UID: "b48999ea-6a3b-427c-a22b-621a5cc997db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:05:31.654014 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.654002 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-tmp-dir" pod "cf9455eb-457d-4fec-aca7-f55202209d2c" (UID: "cf9455eb-457d-4fec-aca7-f55202209d2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:05:31.654192 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.654170 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-model-cache" pod "cf9455eb-457d-4fec-aca7-f55202209d2c" (UID: "cf9455eb-457d-4fec-aca7-f55202209d2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:05:31.654611 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.654581 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-tmp-dir" pod "b48999ea-6a3b-427c-a22b-621a5cc997db" (UID: "b48999ea-6a3b-427c-a22b-621a5cc997db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:05:31.654723 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.654652 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-model-cache" pod "b48999ea-6a3b-427c-a22b-621a5cc997db" (UID: "b48999ea-6a3b-427c-a22b-621a5cc997db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:05:31.655117 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.655096 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-home" pod "cf9455eb-457d-4fec-aca7-f55202209d2c" (UID: "cf9455eb-457d-4fec-aca7-f55202209d2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:05:31.657488 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.657462 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf9455eb-457d-4fec-aca7-f55202209d2c-tls-certs" pod "cf9455eb-457d-4fec-aca7-f55202209d2c" (UID: "cf9455eb-457d-4fec-aca7-f55202209d2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:05:31.657582 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.657560 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b48999ea-6a3b-427c-a22b-621a5cc997db-tls-certs" pod "b48999ea-6a3b-427c-a22b-621a5cc997db" (UID: "b48999ea-6a3b-427c-a22b-621a5cc997db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:05:31.657695 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.657661 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-dshm" pod "b48999ea-6a3b-427c-a22b-621a5cc997db" (UID: "b48999ea-6a3b-427c-a22b-621a5cc997db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:05:31.658661 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.658637 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48999ea-6a3b-427c-a22b-621a5cc997db-kube-api-access-h99jc" pod "b48999ea-6a3b-427c-a22b-621a5cc997db" (UID: "b48999ea-6a3b-427c-a22b-621a5cc997db"). InnerVolumeSpecName "kube-api-access-h99jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:05:31.658772 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.658757 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-dshm" pod "cf9455eb-457d-4fec-aca7-f55202209d2c" (UID: "cf9455eb-457d-4fec-aca7-f55202209d2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:05:31.659177 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.659148 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9455eb-457d-4fec-aca7-f55202209d2c-kube-api-access-gq9bg" pod "cf9455eb-457d-4fec-aca7-f55202209d2c" (UID: "cf9455eb-457d-4fec-aca7-f55202209d2c"). InnerVolumeSpecName "kube-api-access-gq9bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:05:31.754514 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.754468 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:31.754514 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.754522 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:31.754788 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.754558 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:31.754788 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.754569 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf9455eb-457d-4fec-aca7-f55202209d2c-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:31.754788 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.754583 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq9bg\" (UniqueName: \"kubernetes.io/projected/cf9455eb-457d-4fec-aca7-f55202209d2c-kube-api-access-gq9bg\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:31.754788 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.754609 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:31.754788 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.754620 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b48999ea-6a3b-427c-a22b-621a5cc997db-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:31.754788 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.754631 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:31.754788 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.754644 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:31.754788 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.754655 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h99jc\" (UniqueName: \"kubernetes.io/projected/b48999ea-6a3b-427c-a22b-621a5cc997db-kube-api-access-h99jc\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:31.754788 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.754677 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b48999ea-6a3b-427c-a22b-621a5cc997db-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:31.754788 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:31.754690 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf9455eb-457d-4fec-aca7-f55202209d2c-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:05:32.181592 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:32.181510 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" Nov 03 22:05:32.181592 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:32.181548 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" event={"ID":"cf9455eb-457d-4fec-aca7-f55202209d2c","Type":"ContainerDied","Data":"a812ebc08aeee073045a89e0f499434114f1ab3cf43c95a288fc9aa6c3357efb"} Nov 03 22:05:32.181837 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:32.181602 4963 scope.go:122] "RemoveContainer" containerID="62952a2812750d957e5b0940b6de5bdaf5856b9cec924dd06ce80c9f88b18b4d" Nov 03 22:05:32.183241 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:32.183220 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" event={"ID":"b48999ea-6a3b-427c-a22b-621a5cc997db","Type":"ContainerDied","Data":"55d3c95afb9d592afe7693d7229881fe7042bb186797a8beee0d7ac33b00793d"} Nov 03 22:05:32.183330 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:32.183243 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9" Nov 03 22:05:32.192819 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:32.192798 4963 scope.go:122] "RemoveContainer" containerID="9ad65fc1ac7e589fd7b5fa327835326ba809b6c7d0df78e390b21a087c37b34c" Nov 03 22:05:32.343405 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:32.343370 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55"] Nov 03 22:05:32.390873 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:32.390850 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55"] Nov 03 22:05:32.428452 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:32.428406 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-791c32/simulator-w0-791c32-kserve-5965f5b9c6-tbf55" podUID="cf9455eb-457d-4fec-aca7-f55202209d2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:05:32.454116 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:32.454079 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9"] Nov 03 22:05:32.464606 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:32.464495 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/distinct-w0-791c32-kserve-78c78966dd-mmzm9"] Nov 03 22:05:32.830799 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:32.830699 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:32.970239 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:32.970197 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b48999ea-6a3b-427c-a22b-621a5cc997db" path="/var/lib/kubelet/pods/b48999ea-6a3b-427c-a22b-621a5cc997db/volumes" Nov 03 22:05:32.970714 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:32.970690 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf9455eb-457d-4fec-aca7-f55202209d2c" path="/var/lib/kubelet/pods/cf9455eb-457d-4fec-aca7-f55202209d2c/volumes" Nov 03 22:05:33.508849 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:33.508812 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:36.626854 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:36.626820 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:38.109771 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:38.109741 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:38.354009 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:38.353974 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:42.528207 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:05:42.528169 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:05:42.707154 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:42.707125 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:47.216566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:47.216535 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:47.497287 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:47.497203 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:51.793642 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:51.793605 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:54.190806 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:54.190766 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:54.375180 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:54.375141 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:54.568723 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:54.568694 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:55.318158 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:55.318121 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:55.501318 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:55.501282 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:55.754859 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:55.754828 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:57.400963 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:57.400931 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:05:57.601137 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:05:57.601107 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:00.559788 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:00.559743 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:01.153891 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:01.153857 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:01.356328 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:01.356291 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:02.162688 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:02.162652 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:02.687199 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:02.687158 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:03.193169 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:03.193127 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd"] Nov 03 22:06:03.193739 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:03.193386 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" podUID="eda7b470-9598-458c-a270-4b0789924f39" containerName="istio-proxy" containerID="cri-o://14f2b039483f8e137cd328724290d890cbd3bfbd8a5b1071872221c50b989b90" gracePeriod=30 Nov 03 22:06:03.221276 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:03.221230 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:07.280421 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:07.280384 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:08.432956 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.432896 4963 generic.go:356] "Generic (PLEG): container finished" podID="eda7b470-9598-458c-a270-4b0789924f39" containerID="14f2b039483f8e137cd328724290d890cbd3bfbd8a5b1071872221c50b989b90" exitCode=0 Nov 03 22:06:08.433362 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.432946 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" event={"ID":"eda7b470-9598-458c-a270-4b0789924f39","Type":"ContainerDied","Data":"14f2b039483f8e137cd328724290d890cbd3bfbd8a5b1071872221c50b989b90"} Nov 03 22:06:08.473697 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.473625 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 22:06:08.478424 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.478396 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eda7b470-9598-458c-a270-4b0789924f39-istio-token\" (UniqueName: \"kubernetes.io/projected/eda7b470-9598-458c-a270-4b0789924f39-istio-token\") pod \"eda7b470-9598-458c-a270-4b0789924f39\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " Nov 03 22:06:08.478601 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.478477 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-workload-certs\") pod \"eda7b470-9598-458c-a270-4b0789924f39\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " Nov 03 22:06:08.478601 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.478519 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eda7b470-9598-458c-a270-4b0789924f39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda7b470-9598-458c-a270-4b0789924f39-istiod-ca-cert\") pod \"eda7b470-9598-458c-a270-4b0789924f39\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " Nov 03 22:06:08.478601 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.478549 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eda7b470-9598-458c-a270-4b0789924f39-kube-api-access-bp5rs\" (UniqueName: \"kubernetes.io/projected/eda7b470-9598-458c-a270-4b0789924f39-kube-api-access-bp5rs\") pod \"eda7b470-9598-458c-a270-4b0789924f39\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " Nov 03 22:06:08.478601 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.478580 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-istio-envoy\") pod \"eda7b470-9598-458c-a270-4b0789924f39\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " Nov 03 22:06:08.478601 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.478604 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-credential-socket\") pod \"eda7b470-9598-458c-a270-4b0789924f39\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " Nov 03 22:06:08.478888 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.478636 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eda7b470-9598-458c-a270-4b0789924f39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda7b470-9598-458c-a270-4b0789924f39-istio-podinfo\") pod \"eda7b470-9598-458c-a270-4b0789924f39\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " Nov 03 22:06:08.478888 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.478665 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-workload-socket\") pod \"eda7b470-9598-458c-a270-4b0789924f39\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " Nov 03 22:06:08.478888 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.478758 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-istio-data\") pod \"eda7b470-9598-458c-a270-4b0789924f39\" (UID: \"eda7b470-9598-458c-a270-4b0789924f39\") " Nov 03 22:06:08.478888 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.478787 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-workload-certs" pod "eda7b470-9598-458c-a270-4b0789924f39" (UID: "eda7b470-9598-458c-a270-4b0789924f39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:08.479108 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.478996 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda7b470-9598-458c-a270-4b0789924f39-istiod-ca-cert" pod "eda7b470-9598-458c-a270-4b0789924f39" (UID: "eda7b470-9598-458c-a270-4b0789924f39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:06:08.479218 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.479183 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-credential-socket" pod "eda7b470-9598-458c-a270-4b0789924f39" (UID: "eda7b470-9598-458c-a270-4b0789924f39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:08.479290 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.479232 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:08.479290 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.479250 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda7b470-9598-458c-a270-4b0789924f39-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:08.479407 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.479290 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-istio-data" pod "eda7b470-9598-458c-a270-4b0789924f39" (UID: "eda7b470-9598-458c-a270-4b0789924f39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:08.479473 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.479403 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-workload-socket" pod "eda7b470-9598-458c-a270-4b0789924f39" (UID: "eda7b470-9598-458c-a270-4b0789924f39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:08.480939 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.480883 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-istio-envoy" pod "eda7b470-9598-458c-a270-4b0789924f39" (UID: "eda7b470-9598-458c-a270-4b0789924f39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:08.481192 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.481169 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eda7b470-9598-458c-a270-4b0789924f39-istio-podinfo" pod "eda7b470-9598-458c-a270-4b0789924f39" (UID: "eda7b470-9598-458c-a270-4b0789924f39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:06:08.481324 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.481309 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda7b470-9598-458c-a270-4b0789924f39-kube-api-access-bp5rs" pod "eda7b470-9598-458c-a270-4b0789924f39" (UID: "eda7b470-9598-458c-a270-4b0789924f39"). InnerVolumeSpecName "kube-api-access-bp5rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:06:08.481387 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.481322 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda7b470-9598-458c-a270-4b0789924f39-istio-token" pod "eda7b470-9598-458c-a270-4b0789924f39" (UID: "eda7b470-9598-458c-a270-4b0789924f39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:06:08.580504 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.580471 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda7b470-9598-458c-a270-4b0789924f39-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:08.580504 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.580502 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp5rs\" (UniqueName: \"kubernetes.io/projected/eda7b470-9598-458c-a270-4b0789924f39-kube-api-access-bp5rs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:08.580734 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.580516 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:08.580734 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.580529 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:08.580734 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.580541 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda7b470-9598-458c-a270-4b0789924f39-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:08.580734 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.580553 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:08.580734 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:08.580564 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda7b470-9598-458c-a270-4b0789924f39-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:09.280459 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:09.280355 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:09.441215 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:09.441184 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" Nov 03 22:06:09.441685 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:09.441209 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd" event={"ID":"eda7b470-9598-458c-a270-4b0789924f39","Type":"ContainerDied","Data":"43564b23f0fc34ded58125f37befb6ec8f47f616344d82a44a43aa10383f923a"} Nov 03 22:06:09.441685 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:09.441273 4963 scope.go:122] "RemoveContainer" containerID="14f2b039483f8e137cd328724290d890cbd3bfbd8a5b1071872221c50b989b90" Nov 03 22:06:09.647127 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:09.647008 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd"] Nov 03 22:06:09.694934 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:09.694883 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-m7fcd"] Nov 03 22:06:10.978768 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:10.978694 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eda7b470-9598-458c-a270-4b0789924f39" path="/var/lib/kubelet/pods/eda7b470-9598-458c-a270-4b0789924f39/volumes" Nov 03 22:06:11.775726 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:11.775687 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:11.961271 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:11.961238 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:12.528567 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:06:12.528527 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:06:15.698638 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:15.698606 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:15.883658 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:15.883619 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:16.069897 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:16.069815 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:16.341247 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:16.341147 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:16.959593 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:16.959553 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:17.144045 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:17.144014 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:21.308317 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:21.308268 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:22.192350 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:22.192316 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:25.805423 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:25.805383 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:29.727729 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:29.727691 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:30.468244 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:30.468201 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:31.371800 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:31.371755 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:33.974238 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:33.974200 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw"] Nov 03 22:06:34.002605 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:34.002574 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:34.187837 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:34.187807 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:35.978622 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:35.978586 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:37.648198 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:37.648161 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:37.835639 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:37.835608 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:39.216043 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:39.216006 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:39.705380 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:39.705346 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:40.081194 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:40.081108 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:40.280666 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:40.280633 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:40.483271 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:40.483234 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:40.675497 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:40.675465 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:40.958470 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:40.958436 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:42.528621 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:06:42.528576 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:06:45.493216 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:45.493178 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:45.684447 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:45.684413 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:47.836859 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:47.836814 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:48.995726 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:48.995682 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" podUID="7eef8d7a-0034-40d0-9027-d6c4528dd24e" containerName="main" containerID="cri-o://bbca12f72451053ee1650ee2c7804197fa2f0988ae39a577274c02736b4370cb" gracePeriod=45 Nov 03 22:06:49.265567 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.265534 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 22:06:49.345538 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.345501 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-tmp-dir\") pod \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " Nov 03 22:06:49.345716 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.345543 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eef8d7a-0034-40d0-9027-d6c4528dd24e-tls-certs\" (UniqueName: \"kubernetes.io/secret/7eef8d7a-0034-40d0-9027-d6c4528dd24e-tls-certs\") pod \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " Nov 03 22:06:49.345716 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.345609 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-home\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-home\") pod \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " Nov 03 22:06:49.345716 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.345643 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eef8d7a-0034-40d0-9027-d6c4528dd24e-kube-api-access-vsjk4\" (UniqueName: \"kubernetes.io/projected/7eef8d7a-0034-40d0-9027-d6c4528dd24e-kube-api-access-vsjk4\") pod \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " Nov 03 22:06:49.345716 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.345691 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-dshm\") pod \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " Nov 03 22:06:49.345971 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.345750 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-model-cache\") pod \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\" (UID: \"7eef8d7a-0034-40d0-9027-d6c4528dd24e\") " Nov 03 22:06:49.345971 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.345869 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-tmp-dir" pod "7eef8d7a-0034-40d0-9027-d6c4528dd24e" (UID: "7eef8d7a-0034-40d0-9027-d6c4528dd24e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:49.345971 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.345892 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-home" pod "7eef8d7a-0034-40d0-9027-d6c4528dd24e" (UID: "7eef8d7a-0034-40d0-9027-d6c4528dd24e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:49.346199 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.346174 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-model-cache" pod "7eef8d7a-0034-40d0-9027-d6c4528dd24e" (UID: "7eef8d7a-0034-40d0-9027-d6c4528dd24e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:49.346343 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.346313 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:49.346413 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.346333 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:49.346413 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.346360 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:49.348003 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.347979 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-dshm" pod "7eef8d7a-0034-40d0-9027-d6c4528dd24e" (UID: "7eef8d7a-0034-40d0-9027-d6c4528dd24e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:06:49.348137 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.348118 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eef8d7a-0034-40d0-9027-d6c4528dd24e-tls-certs" pod "7eef8d7a-0034-40d0-9027-d6c4528dd24e" (UID: "7eef8d7a-0034-40d0-9027-d6c4528dd24e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:06:49.348201 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.348125 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eef8d7a-0034-40d0-9027-d6c4528dd24e-kube-api-access-vsjk4" pod "7eef8d7a-0034-40d0-9027-d6c4528dd24e" (UID: "7eef8d7a-0034-40d0-9027-d6c4528dd24e"). InnerVolumeSpecName "kube-api-access-vsjk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:06:49.447821 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.447768 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eef8d7a-0034-40d0-9027-d6c4528dd24e-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:49.447821 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.447816 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsjk4\" (UniqueName: \"kubernetes.io/projected/7eef8d7a-0034-40d0-9027-d6c4528dd24e-kube-api-access-vsjk4\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:49.447821 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.447827 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eef8d7a-0034-40d0-9027-d6c4528dd24e-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:06:49.691694 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.691116 4963 generic.go:356] "Generic (PLEG): container finished" podID="7eef8d7a-0034-40d0-9027-d6c4528dd24e" containerID="bbca12f72451053ee1650ee2c7804197fa2f0988ae39a577274c02736b4370cb" exitCode=0 Nov 03 22:06:49.691694 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.691196 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" event={"ID":"7eef8d7a-0034-40d0-9027-d6c4528dd24e","Type":"ContainerDied","Data":"bbca12f72451053ee1650ee2c7804197fa2f0988ae39a577274c02736b4370cb"} Nov 03 22:06:49.691694 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.691221 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" Nov 03 22:06:49.691694 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.691240 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw" event={"ID":"7eef8d7a-0034-40d0-9027-d6c4528dd24e","Type":"ContainerDied","Data":"5d09ac227d3ed290dfeec9b46c3ea42e9260b0e3a042655374df5fa14edc3ea0"} Nov 03 22:06:49.691694 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.691265 4963 scope.go:122] "RemoveContainer" containerID="bbca12f72451053ee1650ee2c7804197fa2f0988ae39a577274c02736b4370cb" Nov 03 22:06:49.703876 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.703849 4963 scope.go:122] "RemoveContainer" containerID="bbca12f72451053ee1650ee2c7804197fa2f0988ae39a577274c02736b4370cb" Nov 03 22:06:49.704242 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:06:49.704212 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbca12f72451053ee1650ee2c7804197fa2f0988ae39a577274c02736b4370cb\": container with ID starting with bbca12f72451053ee1650ee2c7804197fa2f0988ae39a577274c02736b4370cb not found: ID does not exist" containerID="bbca12f72451053ee1650ee2c7804197fa2f0988ae39a577274c02736b4370cb" Nov 03 22:06:49.704333 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.704255 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbca12f72451053ee1650ee2c7804197fa2f0988ae39a577274c02736b4370cb"} err="failed to get container status \"bbca12f72451053ee1650ee2c7804197fa2f0988ae39a577274c02736b4370cb\": rpc error: code = NotFound desc = could not find container \"bbca12f72451053ee1650ee2c7804197fa2f0988ae39a577274c02736b4370cb\": container with ID starting with bbca12f72451053ee1650ee2c7804197fa2f0988ae39a577274c02736b4370cb not found: ID does not exist" Nov 03 22:06:49.852422 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.852392 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw"] Nov 03 22:06:49.900083 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:49.900044 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9818d8/rate-test-model-9818d8-kserve-69b8cc59d5-xldvw"] Nov 03 22:06:50.206705 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:50.206675 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:50.977000 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:50.976965 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eef8d7a-0034-40d0-9027-d6c4528dd24e" path="/var/lib/kubelet/pods/7eef8d7a-0034-40d0-9027-d6c4528dd24e/volumes" Nov 03 22:06:51.941924 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:51.941876 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:52.316889 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:52.316800 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:52.506941 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:52.506872 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:55.005469 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:55.005363 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:57.874871 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:57.874841 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:06:59.728523 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:06:59.728493 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:04.220418 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:04.220314 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:04.973500 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:04.973453 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:08.899531 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:08.899472 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:11.198129 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:11.198088 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:12.239397 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:12.239366 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:12.428631 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:12.428595 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:12.528995 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:07:12.528894 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:07:12.694321 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:12.694292 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:13.644443 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:13.644397 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:17.228410 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:17.228369 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:17.778534 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:17.778500 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z"] Nov 03 22:07:17.778737 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:17.778716 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" podUID="0705edee-7b08-4a0d-b47c-7b8ce7f3279a" containerName="maas-api" containerID="cri-o://0726097228a71e7cc1704fa2f0c4b6a99773034a443e976df0453921b0c032a7" gracePeriod=30 Nov 03 22:07:17.945438 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:17.945404 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:18.064492 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.064461 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 22:07:18.146098 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.146052 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-openshift-service-ca\") pod \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\" (UID: \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\") " Nov 03 22:07:18.146297 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.146149 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-metrics-certs\") pod \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\" (UID: \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\") " Nov 03 22:07:18.146297 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.146183 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-kube-api-access-7jwz8\" (UniqueName: \"kubernetes.io/projected/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-kube-api-access-7jwz8\") pod \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\" (UID: \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\") " Nov 03 22:07:18.146297 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.146258 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-maas-api-tls\") pod \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\" (UID: \"0705edee-7b08-4a0d-b47c-7b8ce7f3279a\") " Nov 03 22:07:18.146504 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.146474 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-openshift-service-ca" pod "0705edee-7b08-4a0d-b47c-7b8ce7f3279a" (UID: "0705edee-7b08-4a0d-b47c-7b8ce7f3279a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:07:18.148484 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.148454 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-maas-api-tls" pod "0705edee-7b08-4a0d-b47c-7b8ce7f3279a" (UID: "0705edee-7b08-4a0d-b47c-7b8ce7f3279a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:07:18.148627 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.148482 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-metrics-certs" pod "0705edee-7b08-4a0d-b47c-7b8ce7f3279a" (UID: "0705edee-7b08-4a0d-b47c-7b8ce7f3279a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:07:18.148627 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.148482 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-kube-api-access-7jwz8" pod "0705edee-7b08-4a0d-b47c-7b8ce7f3279a" (UID: "0705edee-7b08-4a0d-b47c-7b8ce7f3279a"). InnerVolumeSpecName "kube-api-access-7jwz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:07:18.247580 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.247544 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:18.247580 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.247577 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jwz8\" (UniqueName: \"kubernetes.io/projected/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-kube-api-access-7jwz8\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:18.247580 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.247587 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:18.248090 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.247596 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0705edee-7b08-4a0d-b47c-7b8ce7f3279a-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:18.869168 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.869136 4963 generic.go:356] "Generic (PLEG): container finished" podID="0705edee-7b08-4a0d-b47c-7b8ce7f3279a" containerID="0726097228a71e7cc1704fa2f0c4b6a99773034a443e976df0453921b0c032a7" exitCode=0 Nov 03 22:07:18.869396 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.869188 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" Nov 03 22:07:18.869396 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.869226 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" event={"ID":"0705edee-7b08-4a0d-b47c-7b8ce7f3279a","Type":"ContainerDied","Data":"0726097228a71e7cc1704fa2f0c4b6a99773034a443e976df0453921b0c032a7"} Nov 03 22:07:18.869396 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.869291 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z" event={"ID":"0705edee-7b08-4a0d-b47c-7b8ce7f3279a","Type":"ContainerDied","Data":"93e1fb3ba409e46b2298aa8930c9a6103225809a8b85bec2a2bb08d124bbb9c9"} Nov 03 22:07:18.869396 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.869324 4963 scope.go:122] "RemoveContainer" containerID="0726097228a71e7cc1704fa2f0c4b6a99773034a443e976df0453921b0c032a7" Nov 03 22:07:18.879723 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.879705 4963 scope.go:122] "RemoveContainer" containerID="0726097228a71e7cc1704fa2f0c4b6a99773034a443e976df0453921b0c032a7" Nov 03 22:07:18.880065 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:07:18.880044 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0726097228a71e7cc1704fa2f0c4b6a99773034a443e976df0453921b0c032a7\": container with ID starting with 0726097228a71e7cc1704fa2f0c4b6a99773034a443e976df0453921b0c032a7 not found: ID does not exist" containerID="0726097228a71e7cc1704fa2f0c4b6a99773034a443e976df0453921b0c032a7" Nov 03 22:07:18.880135 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:18.880073 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0726097228a71e7cc1704fa2f0c4b6a99773034a443e976df0453921b0c032a7"} err="failed to get container status \"0726097228a71e7cc1704fa2f0c4b6a99773034a443e976df0453921b0c032a7\": rpc error: code = NotFound desc = could not find container \"0726097228a71e7cc1704fa2f0c4b6a99773034a443e976df0453921b0c032a7\": container with ID starting with 0726097228a71e7cc1704fa2f0c4b6a99773034a443e976df0453921b0c032a7 not found: ID does not exist" Nov 03 22:07:19.026683 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:19.026644 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z"] Nov 03 22:07:19.074141 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:19.074104 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-714979-665cc98d97-9mn9z"] Nov 03 22:07:19.626958 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:19.626897 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:20.967209 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:20.967178 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0705edee-7b08-4a0d-b47c-7b8ce7f3279a" path="/var/lib/kubelet/pods/0705edee-7b08-4a0d-b47c-7b8ce7f3279a/volumes" Nov 03 22:07:22.197764 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:22.197724 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:22.396531 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:22.396495 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:23.749265 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:23.749230 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:24.300603 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:24.300570 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:24.493700 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:24.493667 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:24.741500 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:24.741458 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:28.642520 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:28.642478 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs"] Nov 03 22:07:28.645288 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:28.642752 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" podUID="e33113d6-3645-47a5-8f09-edee50ffb54d" containerName="maas-api" containerID="cri-o://bf251979acefd9730c0ed879339e05aa2607a4a4a43aa94b4addb9911e71af38" gracePeriod=30 Nov 03 22:07:28.802094 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:28.802058 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:28.931242 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:28.931205 4963 generic.go:356] "Generic (PLEG): container finished" podID="e33113d6-3645-47a5-8f09-edee50ffb54d" containerID="bf251979acefd9730c0ed879339e05aa2607a4a4a43aa94b4addb9911e71af38" exitCode=0 Nov 03 22:07:28.931412 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:28.931269 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" event={"ID":"e33113d6-3645-47a5-8f09-edee50ffb54d","Type":"ContainerDied","Data":"bf251979acefd9730c0ed879339e05aa2607a4a4a43aa94b4addb9911e71af38"} Nov 03 22:07:28.931412 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:28.931324 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" event={"ID":"e33113d6-3645-47a5-8f09-edee50ffb54d","Type":"ContainerDied","Data":"ddc9a6cdc07c47e834867987e6783dba0278751183cf5064c9c8a97f9ffe94d1"} Nov 03 22:07:28.931412 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:28.931341 4963 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddc9a6cdc07c47e834867987e6783dba0278751183cf5064c9c8a97f9ffe94d1" Nov 03 22:07:28.955191 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:28.955161 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 22:07:29.067793 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:29.067438 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e33113d6-3645-47a5-8f09-edee50ffb54d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e33113d6-3645-47a5-8f09-edee50ffb54d-maas-api-tls\") pod \"e33113d6-3645-47a5-8f09-edee50ffb54d\" (UID: \"e33113d6-3645-47a5-8f09-edee50ffb54d\") " Nov 03 22:07:29.067793 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:29.067615 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e33113d6-3645-47a5-8f09-edee50ffb54d-kube-api-access-xfj2k\" (UniqueName: \"kubernetes.io/projected/e33113d6-3645-47a5-8f09-edee50ffb54d-kube-api-access-xfj2k\") pod \"e33113d6-3645-47a5-8f09-edee50ffb54d\" (UID: \"e33113d6-3645-47a5-8f09-edee50ffb54d\") " Nov 03 22:07:29.067793 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:29.067703 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e33113d6-3645-47a5-8f09-edee50ffb54d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e33113d6-3645-47a5-8f09-edee50ffb54d-metrics-certs\") pod \"e33113d6-3645-47a5-8f09-edee50ffb54d\" (UID: \"e33113d6-3645-47a5-8f09-edee50ffb54d\") " Nov 03 22:07:29.067793 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:29.067754 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e33113d6-3645-47a5-8f09-edee50ffb54d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e33113d6-3645-47a5-8f09-edee50ffb54d-openshift-service-ca\") pod \"e33113d6-3645-47a5-8f09-edee50ffb54d\" (UID: \"e33113d6-3645-47a5-8f09-edee50ffb54d\") " Nov 03 22:07:29.068606 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:29.068563 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e33113d6-3645-47a5-8f09-edee50ffb54d-openshift-service-ca" pod "e33113d6-3645-47a5-8f09-edee50ffb54d" (UID: "e33113d6-3645-47a5-8f09-edee50ffb54d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:07:29.069707 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:29.069536 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e33113d6-3645-47a5-8f09-edee50ffb54d-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:29.072203 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:29.071207 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33113d6-3645-47a5-8f09-edee50ffb54d-kube-api-access-xfj2k" pod "e33113d6-3645-47a5-8f09-edee50ffb54d" (UID: "e33113d6-3645-47a5-8f09-edee50ffb54d"). InnerVolumeSpecName "kube-api-access-xfj2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:07:29.072203 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:29.072154 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e33113d6-3645-47a5-8f09-edee50ffb54d-maas-api-tls" pod "e33113d6-3645-47a5-8f09-edee50ffb54d" (UID: "e33113d6-3645-47a5-8f09-edee50ffb54d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:07:29.075018 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:29.074991 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e33113d6-3645-47a5-8f09-edee50ffb54d-metrics-certs" pod "e33113d6-3645-47a5-8f09-edee50ffb54d" (UID: "e33113d6-3645-47a5-8f09-edee50ffb54d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:07:29.171480 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:29.171391 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfj2k\" (UniqueName: \"kubernetes.io/projected/e33113d6-3645-47a5-8f09-edee50ffb54d-kube-api-access-xfj2k\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:29.171480 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:29.171424 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e33113d6-3645-47a5-8f09-edee50ffb54d-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:29.171480 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:29.171433 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e33113d6-3645-47a5-8f09-edee50ffb54d-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:29.657037 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:29.657002 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx"] Nov 03 22:07:29.667048 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:29.667020 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb"] Nov 03 22:07:29.790953 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:29.790897 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:29.937313 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:29.937215 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs" Nov 03 22:07:30.096251 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:30.096216 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs"] Nov 03 22:07:30.098533 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:30.098505 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9818d8-86bf67cbc6-pkcrs"] Nov 03 22:07:30.533640 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:30.533608 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:30.968473 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:30.968440 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e33113d6-3645-47a5-8f09-edee50ffb54d" path="/var/lib/kubelet/pods/e33113d6-3645-47a5-8f09-edee50ffb54d/volumes" Nov 03 22:07:31.621919 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:31.621864 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:31.625129 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:31.625101 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 22:07:31.658595 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:31.658566 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 22:07:32.406271 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:32.406238 4963 scope.go:122] "RemoveContainer" containerID="bf251979acefd9730c0ed879339e05aa2607a4a4a43aa94b4addb9911e71af38" Nov 03 22:07:32.557231 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:32.557197 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:32.744328 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:32.744293 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:34.093060 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:34.093030 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:34.593042 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:34.593012 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:35.345623 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:35.345581 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:35.545842 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:35.545800 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:37.914813 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:37.914783 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:38.664014 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:38.663980 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:38.846533 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:38.846499 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:40.252326 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:40.252243 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:41.483449 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:41.483402 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:42.403163 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:42.403127 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:42.528476 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:07:42.528431 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:07:42.603354 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:42.603318 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:43.858633 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:43.858584 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:44.058713 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:44.058676 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:44.677864 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:44.677828 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" podUID="7bdfcd76-3dd3-46fd-a382-0e40e648aa17" containerName="main" containerID="cri-o://fdd44493093f157a65556a76d1127e2789c257d4b59ef446f464121da90df6af" gracePeriod=45 Nov 03 22:07:44.683479 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:44.683450 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:44.686950 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:44.686887 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" podUID="662bceeb-beae-4a4f-b08a-08706bed2d2e" containerName="main" containerID="cri-o://59ac55ed0c753f0d0d5252a44940ff4fd633caa838c1a0508918a3c56b259ab7" gracePeriod=45 Nov 03 22:07:44.860890 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:44.859343 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" podUID="662bceeb-beae-4a4f-b08a-08706bed2d2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": dial tcp 10.132.0.63:8000: connect: connection refused" Nov 03 22:07:44.961138 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:44.961106 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 22:07:44.989652 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:44.989624 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 22:07:45.027442 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.027403 4963 generic.go:356] "Generic (PLEG): container finished" podID="7bdfcd76-3dd3-46fd-a382-0e40e648aa17" containerID="fdd44493093f157a65556a76d1127e2789c257d4b59ef446f464121da90df6af" exitCode=0 Nov 03 22:07:45.027655 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.027455 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" Nov 03 22:07:45.027655 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.027486 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" event={"ID":"7bdfcd76-3dd3-46fd-a382-0e40e648aa17","Type":"ContainerDied","Data":"fdd44493093f157a65556a76d1127e2789c257d4b59ef446f464121da90df6af"} Nov 03 22:07:45.027655 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.027527 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx" event={"ID":"7bdfcd76-3dd3-46fd-a382-0e40e648aa17","Type":"ContainerDied","Data":"7fac3fbd8fce1fc2bd3d39b8550e854eb7b87d44837369246a1bebd15d11fce0"} Nov 03 22:07:45.027655 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.027552 4963 scope.go:122] "RemoveContainer" containerID="fdd44493093f157a65556a76d1127e2789c257d4b59ef446f464121da90df6af" Nov 03 22:07:45.029435 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.029405 4963 generic.go:356] "Generic (PLEG): container finished" podID="662bceeb-beae-4a4f-b08a-08706bed2d2e" containerID="59ac55ed0c753f0d0d5252a44940ff4fd633caa838c1a0508918a3c56b259ab7" exitCode=0 Nov 03 22:07:45.029568 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.029464 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" Nov 03 22:07:45.029568 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.029501 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" event={"ID":"662bceeb-beae-4a4f-b08a-08706bed2d2e","Type":"ContainerDied","Data":"59ac55ed0c753f0d0d5252a44940ff4fd633caa838c1a0508918a3c56b259ab7"} Nov 03 22:07:45.029568 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.029549 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb" event={"ID":"662bceeb-beae-4a4f-b08a-08706bed2d2e","Type":"ContainerDied","Data":"3d53d4987aebb33118e8a4cb112925c83c1bf1ee6edd7c9bb1a25e0581ee64a8"} Nov 03 22:07:45.038003 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.037984 4963 scope.go:122] "RemoveContainer" containerID="fdd44493093f157a65556a76d1127e2789c257d4b59ef446f464121da90df6af" Nov 03 22:07:45.038305 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:07:45.038276 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd44493093f157a65556a76d1127e2789c257d4b59ef446f464121da90df6af\": container with ID starting with fdd44493093f157a65556a76d1127e2789c257d4b59ef446f464121da90df6af not found: ID does not exist" containerID="fdd44493093f157a65556a76d1127e2789c257d4b59ef446f464121da90df6af" Nov 03 22:07:45.038363 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.038319 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd44493093f157a65556a76d1127e2789c257d4b59ef446f464121da90df6af"} err="failed to get container status \"fdd44493093f157a65556a76d1127e2789c257d4b59ef446f464121da90df6af\": rpc error: code = NotFound desc = could not find container \"fdd44493093f157a65556a76d1127e2789c257d4b59ef446f464121da90df6af\": container with ID starting with fdd44493093f157a65556a76d1127e2789c257d4b59ef446f464121da90df6af not found: ID does not exist" Nov 03 22:07:45.038363 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.038337 4963 scope.go:122] "RemoveContainer" containerID="59ac55ed0c753f0d0d5252a44940ff4fd633caa838c1a0508918a3c56b259ab7" Nov 03 22:07:45.047314 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.047294 4963 scope.go:122] "RemoveContainer" containerID="59ac55ed0c753f0d0d5252a44940ff4fd633caa838c1a0508918a3c56b259ab7" Nov 03 22:07:45.047591 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:07:45.047568 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ac55ed0c753f0d0d5252a44940ff4fd633caa838c1a0508918a3c56b259ab7\": container with ID starting with 59ac55ed0c753f0d0d5252a44940ff4fd633caa838c1a0508918a3c56b259ab7 not found: ID does not exist" containerID="59ac55ed0c753f0d0d5252a44940ff4fd633caa838c1a0508918a3c56b259ab7" Nov 03 22:07:45.047657 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.047606 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ac55ed0c753f0d0d5252a44940ff4fd633caa838c1a0508918a3c56b259ab7"} err="failed to get container status \"59ac55ed0c753f0d0d5252a44940ff4fd633caa838c1a0508918a3c56b259ab7\": rpc error: code = NotFound desc = could not find container \"59ac55ed0c753f0d0d5252a44940ff4fd633caa838c1a0508918a3c56b259ab7\": container with ID starting with 59ac55ed0c753f0d0d5252a44940ff4fd633caa838c1a0508918a3c56b259ab7 not found: ID does not exist" Nov 03 22:07:45.143760 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.143719 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-kube-api-access-x6ksh\" (UniqueName: \"kubernetes.io/projected/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-kube-api-access-x6ksh\") pod \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " Nov 03 22:07:45.143963 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.143783 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-model-cache\") pod \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " Nov 03 22:07:45.144044 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.143979 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-home\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-home\") pod \"662bceeb-beae-4a4f-b08a-08706bed2d2e\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " Nov 03 22:07:45.144102 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.144068 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-dshm\") pod \"662bceeb-beae-4a4f-b08a-08706bed2d2e\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " Nov 03 22:07:45.144102 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.144080 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-model-cache" pod "7bdfcd76-3dd3-46fd-a382-0e40e648aa17" (UID: "7bdfcd76-3dd3-46fd-a382-0e40e648aa17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:45.144218 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.144113 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-tmp-dir\") pod \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " Nov 03 22:07:45.144218 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.144154 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/662bceeb-beae-4a4f-b08a-08706bed2d2e-kube-api-access-98p6z\" (UniqueName: \"kubernetes.io/projected/662bceeb-beae-4a4f-b08a-08706bed2d2e-kube-api-access-98p6z\") pod \"662bceeb-beae-4a4f-b08a-08706bed2d2e\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " Nov 03 22:07:45.144218 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.144182 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-home\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-home\") pod \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " Nov 03 22:07:45.144218 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.144199 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-home" pod "662bceeb-beae-4a4f-b08a-08706bed2d2e" (UID: "662bceeb-beae-4a4f-b08a-08706bed2d2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:45.144423 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.144238 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-tmp-dir\") pod \"662bceeb-beae-4a4f-b08a-08706bed2d2e\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " Nov 03 22:07:45.144423 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.144331 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-model-cache\") pod \"662bceeb-beae-4a4f-b08a-08706bed2d2e\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " Nov 03 22:07:45.144525 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.144439 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-tmp-dir" pod "7bdfcd76-3dd3-46fd-a382-0e40e648aa17" (UID: "7bdfcd76-3dd3-46fd-a382-0e40e648aa17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:45.144645 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.144624 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-home" pod "7bdfcd76-3dd3-46fd-a382-0e40e648aa17" (UID: "7bdfcd76-3dd3-46fd-a382-0e40e648aa17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:45.144820 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.144801 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/662bceeb-beae-4a4f-b08a-08706bed2d2e-tls-certs\" (UniqueName: \"kubernetes.io/secret/662bceeb-beae-4a4f-b08a-08706bed2d2e-tls-certs\") pod \"662bceeb-beae-4a4f-b08a-08706bed2d2e\" (UID: \"662bceeb-beae-4a4f-b08a-08706bed2d2e\") " Nov 03 22:07:45.144895 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.144835 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-tls-certs\") pod \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " Nov 03 22:07:45.144895 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.144868 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-dshm\") pod \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\" (UID: \"7bdfcd76-3dd3-46fd-a382-0e40e648aa17\") " Nov 03 22:07:45.145150 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.144809 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-tmp-dir" pod "662bceeb-beae-4a4f-b08a-08706bed2d2e" (UID: "662bceeb-beae-4a4f-b08a-08706bed2d2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:45.145543 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.145522 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.145543 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.145544 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.145713 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.145560 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.145713 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.145573 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.145713 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.145585 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.146679 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.145122 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-model-cache" pod "662bceeb-beae-4a4f-b08a-08706bed2d2e" (UID: "662bceeb-beae-4a4f-b08a-08706bed2d2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:45.146959 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.146897 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-dshm" pod "662bceeb-beae-4a4f-b08a-08706bed2d2e" (UID: "662bceeb-beae-4a4f-b08a-08706bed2d2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:45.147340 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.147308 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-dshm" pod "7bdfcd76-3dd3-46fd-a382-0e40e648aa17" (UID: "7bdfcd76-3dd3-46fd-a382-0e40e648aa17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:45.147483 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.147459 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662bceeb-beae-4a4f-b08a-08706bed2d2e-kube-api-access-98p6z" pod "662bceeb-beae-4a4f-b08a-08706bed2d2e" (UID: "662bceeb-beae-4a4f-b08a-08706bed2d2e"). InnerVolumeSpecName "kube-api-access-98p6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:07:45.148833 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.148811 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-kube-api-access-x6ksh" pod "7bdfcd76-3dd3-46fd-a382-0e40e648aa17" (UID: "7bdfcd76-3dd3-46fd-a382-0e40e648aa17"). InnerVolumeSpecName "kube-api-access-x6ksh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:07:45.149080 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.149053 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-tls-certs" pod "7bdfcd76-3dd3-46fd-a382-0e40e648aa17" (UID: "7bdfcd76-3dd3-46fd-a382-0e40e648aa17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:07:45.149194 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.149085 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662bceeb-beae-4a4f-b08a-08706bed2d2e-tls-certs" pod "662bceeb-beae-4a4f-b08a-08706bed2d2e" (UID: "662bceeb-beae-4a4f-b08a-08706bed2d2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:07:45.229232 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.229195 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:45.246254 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.246214 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.246254 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.246254 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98p6z\" (UniqueName: \"kubernetes.io/projected/662bceeb-beae-4a4f-b08a-08706bed2d2e-kube-api-access-98p6z\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.246469 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.246269 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/662bceeb-beae-4a4f-b08a-08706bed2d2e-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.246469 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.246278 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/662bceeb-beae-4a4f-b08a-08706bed2d2e-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.246469 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.246288 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.246469 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.246296 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.246469 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.246304 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6ksh\" (UniqueName: \"kubernetes.io/projected/7bdfcd76-3dd3-46fd-a382-0e40e648aa17-kube-api-access-x6ksh\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:45.492624 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.492521 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb"] Nov 03 22:07:45.502574 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.502537 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/premium-w1-a41ad0-kserve-756f497796-4tttb"] Nov 03 22:07:45.619024 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.618987 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx"] Nov 03 22:07:45.621683 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:45.621658 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a41ad0/distinct-2-w1-a41ad0-kserve-7b89777b88-xtnwx"] Nov 03 22:07:46.966744 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:46.966698 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="662bceeb-beae-4a4f-b08a-08706bed2d2e" path="/var/lib/kubelet/pods/662bceeb-beae-4a4f-b08a-08706bed2d2e/volumes" Nov 03 22:07:46.967355 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:46.967334 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bdfcd76-3dd3-46fd-a382-0e40e648aa17" path="/var/lib/kubelet/pods/7bdfcd76-3dd3-46fd-a382-0e40e648aa17/volumes" Nov 03 22:07:47.703552 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:47.703522 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:48.181877 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:48.181778 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:49.824715 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:49.824681 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:50.308819 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:50.308783 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:52.033890 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:52.033846 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:52.353227 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:52.353111 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:54.430018 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:54.429982 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:55.915276 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:55.915239 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6"] Nov 03 22:07:55.965824 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:55.965785 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:56.283688 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:56.283651 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:56.464832 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:56.464795 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:58.183252 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:58.183217 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:59.045981 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:07:59.045897 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:03.720550 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:03.720513 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:04.382088 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:04.382052 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:07.663552 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:07.663498 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:09.399630 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:09.399599 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:10.935294 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:10.935256 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" podUID="cb3fc56a-e7a0-445a-b206-f903fd3d12d9" containerName="main" containerID="cri-o://daf068cae90b11a3dc4a2baac58dafe535286392b312bd8084329dacf8a34078" gracePeriod=45 Nov 03 22:08:11.183829 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.183798 4963 generic.go:356] "Generic (PLEG): container finished" podID="cb3fc56a-e7a0-445a-b206-f903fd3d12d9" containerID="daf068cae90b11a3dc4a2baac58dafe535286392b312bd8084329dacf8a34078" exitCode=0 Nov 03 22:08:11.184007 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.183866 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" event={"ID":"cb3fc56a-e7a0-445a-b206-f903fd3d12d9","Type":"ContainerDied","Data":"daf068cae90b11a3dc4a2baac58dafe535286392b312bd8084329dacf8a34078"} Nov 03 22:08:11.184007 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.183933 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" event={"ID":"cb3fc56a-e7a0-445a-b206-f903fd3d12d9","Type":"ContainerDied","Data":"19cb79bd2a8d44e1d84f36c3b0906776503f2f684e6e3264aad035416a173f06"} Nov 03 22:08:11.184007 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.183950 4963 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19cb79bd2a8d44e1d84f36c3b0906776503f2f684e6e3264aad035416a173f06" Nov 03 22:08:11.196058 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.196031 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:08:11.323570 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.323541 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-tls-certs\") pod \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " Nov 03 22:08:11.323772 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.323594 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-kube-api-access-mjkpz\" (UniqueName: \"kubernetes.io/projected/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-kube-api-access-mjkpz\") pod \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " Nov 03 22:08:11.323772 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.323663 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-dshm\") pod \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " Nov 03 22:08:11.323921 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.323803 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-tmp-dir\") pod \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " Nov 03 22:08:11.323995 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.323972 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-home\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-home\") pod \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " Nov 03 22:08:11.324068 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.324030 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-model-cache\") pod \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\" (UID: \"cb3fc56a-e7a0-445a-b206-f903fd3d12d9\") " Nov 03 22:08:11.324163 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.324131 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-tmp-dir" pod "cb3fc56a-e7a0-445a-b206-f903fd3d12d9" (UID: "cb3fc56a-e7a0-445a-b206-f903fd3d12d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:11.324423 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.324398 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-model-cache" pod "cb3fc56a-e7a0-445a-b206-f903fd3d12d9" (UID: "cb3fc56a-e7a0-445a-b206-f903fd3d12d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:11.324523 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.324506 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:11.324737 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.324706 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-home" pod "cb3fc56a-e7a0-445a-b206-f903fd3d12d9" (UID: "cb3fc56a-e7a0-445a-b206-f903fd3d12d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:11.326579 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.326539 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-dshm" pod "cb3fc56a-e7a0-445a-b206-f903fd3d12d9" (UID: "cb3fc56a-e7a0-445a-b206-f903fd3d12d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:11.326896 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.326872 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-kube-api-access-mjkpz" pod "cb3fc56a-e7a0-445a-b206-f903fd3d12d9" (UID: "cb3fc56a-e7a0-445a-b206-f903fd3d12d9"). InnerVolumeSpecName "kube-api-access-mjkpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:11.327029 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.326928 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-tls-certs" pod "cb3fc56a-e7a0-445a-b206-f903fd3d12d9" (UID: "cb3fc56a-e7a0-445a-b206-f903fd3d12d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:11.425224 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.425187 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjkpz\" (UniqueName: \"kubernetes.io/projected/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-kube-api-access-mjkpz\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:11.425224 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.425217 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:11.425224 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.425225 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:11.425224 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.425234 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:11.425532 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.425244 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3fc56a-e7a0-445a-b206-f903fd3d12d9-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:11.599696 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.599619 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:11.781932 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:11.781864 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:12.191421 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:12.191386 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6" Nov 03 22:08:12.388987 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:12.388944 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6"] Nov 03 22:08:12.437010 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:12.436979 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c70fca/sub-test-model-c70fca-kserve-5559c94b5-wt7m6"] Nov 03 22:08:12.528184 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:08:12.528146 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:08:12.967430 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:12.967393 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3fc56a-e7a0-445a-b206-f903fd3d12d9" path="/var/lib/kubelet/pods/cb3fc56a-e7a0-445a-b206-f903fd3d12d9/volumes" Nov 03 22:08:13.161628 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:13.161577 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:13.329029 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:13.328941 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b"] Nov 03 22:08:13.329498 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:13.329173 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" podUID="62344e41-1f57-4ca4-9805-668cec58dd3b" containerName="maas-api" containerID="cri-o://e69b59c4261aae23748dc570538e05053e2bc9cde902dba552e80f9dead8beaa" gracePeriod=30 Nov 03 22:08:13.466511 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:13.466469 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:13.635655 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:13.635620 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 22:08:13.725406 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:13.722793 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:13.752736 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:13.752697 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62344e41-1f57-4ca4-9805-668cec58dd3b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62344e41-1f57-4ca4-9805-668cec58dd3b-maas-api-tls\") pod \"62344e41-1f57-4ca4-9805-668cec58dd3b\" (UID: \"62344e41-1f57-4ca4-9805-668cec58dd3b\") " Nov 03 22:08:13.752963 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:13.752755 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62344e41-1f57-4ca4-9805-668cec58dd3b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62344e41-1f57-4ca4-9805-668cec58dd3b-openshift-service-ca\") pod \"62344e41-1f57-4ca4-9805-668cec58dd3b\" (UID: \"62344e41-1f57-4ca4-9805-668cec58dd3b\") " Nov 03 22:08:13.752963 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:13.752940 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62344e41-1f57-4ca4-9805-668cec58dd3b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/62344e41-1f57-4ca4-9805-668cec58dd3b-metrics-certs\") pod \"62344e41-1f57-4ca4-9805-668cec58dd3b\" (UID: \"62344e41-1f57-4ca4-9805-668cec58dd3b\") " Nov 03 22:08:13.753099 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:13.752995 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62344e41-1f57-4ca4-9805-668cec58dd3b-kube-api-access-lwpzv\" (UniqueName: \"kubernetes.io/projected/62344e41-1f57-4ca4-9805-668cec58dd3b-kube-api-access-lwpzv\") pod \"62344e41-1f57-4ca4-9805-668cec58dd3b\" (UID: \"62344e41-1f57-4ca4-9805-668cec58dd3b\") " Nov 03 22:08:13.753219 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:13.753197 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62344e41-1f57-4ca4-9805-668cec58dd3b-openshift-service-ca" pod "62344e41-1f57-4ca4-9805-668cec58dd3b" (UID: "62344e41-1f57-4ca4-9805-668cec58dd3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:13.753395 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:13.753373 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62344e41-1f57-4ca4-9805-668cec58dd3b-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:13.755436 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:13.755407 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62344e41-1f57-4ca4-9805-668cec58dd3b-kube-api-access-lwpzv" pod "62344e41-1f57-4ca4-9805-668cec58dd3b" (UID: "62344e41-1f57-4ca4-9805-668cec58dd3b"). InnerVolumeSpecName "kube-api-access-lwpzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:13.755657 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:13.755627 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62344e41-1f57-4ca4-9805-668cec58dd3b-maas-api-tls" pod "62344e41-1f57-4ca4-9805-668cec58dd3b" (UID: "62344e41-1f57-4ca4-9805-668cec58dd3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:13.756076 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:13.756051 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62344e41-1f57-4ca4-9805-668cec58dd3b-metrics-certs" pod "62344e41-1f57-4ca4-9805-668cec58dd3b" (UID: "62344e41-1f57-4ca4-9805-668cec58dd3b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:13.854357 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:13.854316 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62344e41-1f57-4ca4-9805-668cec58dd3b-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:13.854357 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:13.854351 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwpzv\" (UniqueName: \"kubernetes.io/projected/62344e41-1f57-4ca4-9805-668cec58dd3b-kube-api-access-lwpzv\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:13.854357 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:13.854361 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62344e41-1f57-4ca4-9805-668cec58dd3b-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:14.126191 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:14.126158 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:14.203883 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:14.203843 4963 generic.go:356] "Generic (PLEG): container finished" podID="62344e41-1f57-4ca4-9805-668cec58dd3b" containerID="e69b59c4261aae23748dc570538e05053e2bc9cde902dba552e80f9dead8beaa" exitCode=0 Nov 03 22:08:14.204094 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:14.203895 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" Nov 03 22:08:14.204094 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:14.203954 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" event={"ID":"62344e41-1f57-4ca4-9805-668cec58dd3b","Type":"ContainerDied","Data":"e69b59c4261aae23748dc570538e05053e2bc9cde902dba552e80f9dead8beaa"} Nov 03 22:08:14.204094 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:14.204014 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b" event={"ID":"62344e41-1f57-4ca4-9805-668cec58dd3b","Type":"ContainerDied","Data":"fb4779da15349452dfd1e4910cdf0f37564db733cdbfe1729147ca05fdcbac9d"} Nov 03 22:08:14.204094 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:14.204040 4963 scope.go:122] "RemoveContainer" containerID="e69b59c4261aae23748dc570538e05053e2bc9cde902dba552e80f9dead8beaa" Nov 03 22:08:14.214788 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:14.214767 4963 scope.go:122] "RemoveContainer" containerID="e69b59c4261aae23748dc570538e05053e2bc9cde902dba552e80f9dead8beaa" Nov 03 22:08:14.215118 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:08:14.215096 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69b59c4261aae23748dc570538e05053e2bc9cde902dba552e80f9dead8beaa\": container with ID starting with e69b59c4261aae23748dc570538e05053e2bc9cde902dba552e80f9dead8beaa not found: ID does not exist" containerID="e69b59c4261aae23748dc570538e05053e2bc9cde902dba552e80f9dead8beaa" Nov 03 22:08:14.215213 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:14.215132 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69b59c4261aae23748dc570538e05053e2bc9cde902dba552e80f9dead8beaa"} err="failed to get container status \"e69b59c4261aae23748dc570538e05053e2bc9cde902dba552e80f9dead8beaa\": rpc error: code = NotFound desc = could not find container \"e69b59c4261aae23748dc570538e05053e2bc9cde902dba552e80f9dead8beaa\": container with ID starting with e69b59c4261aae23748dc570538e05053e2bc9cde902dba552e80f9dead8beaa not found: ID does not exist" Nov 03 22:08:14.362684 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:14.362640 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b"] Nov 03 22:08:14.367093 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:14.367061 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a41ad0-68f564578f-29z2b"] Nov 03 22:08:14.975043 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:14.975002 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62344e41-1f57-4ca4-9805-668cec58dd3b" path="/var/lib/kubelet/pods/62344e41-1f57-4ca4-9805-668cec58dd3b/volumes" Nov 03 22:08:15.343777 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:15.343690 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:16.467605 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:16.467574 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:16.651735 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:16.651700 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:17.919482 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:17.919448 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:18.457485 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:18.457407 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:19.049830 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:19.049797 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f"] Nov 03 22:08:19.096728 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:19.096693 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q"] Nov 03 22:08:19.098728 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:19.098701 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x"] Nov 03 22:08:19.306333 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:19.306250 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:19.987294 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:19.987254 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6"] Nov 03 22:08:20.065718 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:20.065671 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:20.265598 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:20.265512 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:20.510894 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:20.510844 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:25.008878 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:25.008843 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:25.346312 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:25.346226 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:26.688777 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:26.688741 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:28.152764 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:28.152725 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:28.356313 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:28.356282 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:29.938693 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:29.938653 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:32.439425 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:32.439392 4963 scope.go:122] "RemoveContainer" containerID="daf068cae90b11a3dc4a2baac58dafe535286392b312bd8084329dacf8a34078" Nov 03 22:08:32.453763 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:32.453728 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:32.979624 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:32.979589 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:33.164493 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:33.164462 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:33.355561 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:33.355450 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:33.964057 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:33.964012 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:34.073292 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.073249 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" podUID="fae009e3-4ccc-46ef-895c-5ecc6ea4e599" containerName="main" containerID="cri-o://9186ab3aaac1096ee0dc3e212754aaef3d9b2c38a0ccec92bd791d20a973697a" gracePeriod=45 Nov 03 22:08:34.117330 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.117290 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" podUID="4524d0a6-727b-47a9-b3b9-e83e5cf3a52d" containerName="main" containerID="cri-o://76861975ac0597c193c769913dfb0fec6dbf2f0e3fbfa1428d069c722139f015" gracePeriod=45 Nov 03 22:08:34.118369 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.118333 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" podUID="fa8baac2-125d-42f6-8449-5fd590bf4b2b" containerName="main" containerID="cri-o://75d48a7aa0c9b43e9d4131786a80a1c965eb47dc57619b8399d71bed70b6c530" gracePeriod=45 Nov 03 22:08:34.323871 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.323195 4963 generic.go:356] "Generic (PLEG): container finished" podID="fa8baac2-125d-42f6-8449-5fd590bf4b2b" containerID="75d48a7aa0c9b43e9d4131786a80a1c965eb47dc57619b8399d71bed70b6c530" exitCode=0 Nov 03 22:08:34.323871 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.323274 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" event={"ID":"fa8baac2-125d-42f6-8449-5fd590bf4b2b","Type":"ContainerDied","Data":"75d48a7aa0c9b43e9d4131786a80a1c965eb47dc57619b8399d71bed70b6c530"} Nov 03 22:08:34.325084 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.325060 4963 generic.go:356] "Generic (PLEG): container finished" podID="fae009e3-4ccc-46ef-895c-5ecc6ea4e599" containerID="9186ab3aaac1096ee0dc3e212754aaef3d9b2c38a0ccec92bd791d20a973697a" exitCode=0 Nov 03 22:08:34.325195 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.325126 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" event={"ID":"fae009e3-4ccc-46ef-895c-5ecc6ea4e599","Type":"ContainerDied","Data":"9186ab3aaac1096ee0dc3e212754aaef3d9b2c38a0ccec92bd791d20a973697a"} Nov 03 22:08:34.326981 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.326955 4963 generic.go:356] "Generic (PLEG): container finished" podID="4524d0a6-727b-47a9-b3b9-e83e5cf3a52d" containerID="76861975ac0597c193c769913dfb0fec6dbf2f0e3fbfa1428d069c722139f015" exitCode=0 Nov 03 22:08:34.327084 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.327009 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" event={"ID":"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d","Type":"ContainerDied","Data":"76861975ac0597c193c769913dfb0fec6dbf2f0e3fbfa1428d069c722139f015"} Nov 03 22:08:34.428844 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.428814 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 22:08:34.432018 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.431993 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 22:08:34.435549 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.435523 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 22:08:34.473314 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.473280 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-tls-certs\" (UniqueName: \"kubernetes.io/secret/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-tls-certs\") pod \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " Nov 03 22:08:34.473314 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.473321 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-dshm\") pod \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " Nov 03 22:08:34.473588 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.473354 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-model-cache\") pod \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " Nov 03 22:08:34.473588 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.473483 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-tls-certs\" (UniqueName: \"kubernetes.io/secret/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-tls-certs\") pod \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " Nov 03 22:08:34.473588 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.473536 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-tmp-dir\") pod \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " Nov 03 22:08:34.473588 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.473569 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-model-cache\") pod \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " Nov 03 22:08:34.473809 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.473613 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-dshm\") pod \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " Nov 03 22:08:34.473809 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.473684 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-model-cache" pod "fae009e3-4ccc-46ef-895c-5ecc6ea4e599" (UID: "fae009e3-4ccc-46ef-895c-5ecc6ea4e599"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:34.473809 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.473688 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-home\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-home\") pod \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " Nov 03 22:08:34.473809 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.473749 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa8baac2-125d-42f6-8449-5fd590bf4b2b-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa8baac2-125d-42f6-8449-5fd590bf4b2b-tls-certs\") pod \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " Nov 03 22:08:34.474423 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.474018 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-kube-api-access-4jw9x\" (UniqueName: \"kubernetes.io/projected/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-kube-api-access-4jw9x\") pod \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " Nov 03 22:08:34.474423 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.474089 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-tmp-dir\") pod \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " Nov 03 22:08:34.474423 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.474154 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-home\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-home\") pod \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " Nov 03 22:08:34.474423 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.474193 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-home\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-home\") pod \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " Nov 03 22:08:34.474423 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.474277 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-model-cache\") pod \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " Nov 03 22:08:34.474423 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.474334 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-dshm\") pod \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " Nov 03 22:08:34.474423 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.474374 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-kube-api-access-npn2l\" (UniqueName: \"kubernetes.io/projected/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-kube-api-access-npn2l\") pod \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\" (UID: \"fae009e3-4ccc-46ef-895c-5ecc6ea4e599\") " Nov 03 22:08:34.474423 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.474018 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-tmp-dir" pod "fa8baac2-125d-42f6-8449-5fd590bf4b2b" (UID: "fa8baac2-125d-42f6-8449-5fd590bf4b2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:34.474895 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.474043 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-model-cache" pod "fa8baac2-125d-42f6-8449-5fd590bf4b2b" (UID: "fa8baac2-125d-42f6-8449-5fd590bf4b2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:34.474895 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.474203 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-home" pod "fa8baac2-125d-42f6-8449-5fd590bf4b2b" (UID: "fa8baac2-125d-42f6-8449-5fd590bf4b2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:34.474895 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.474415 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-tmp-dir" pod "fae009e3-4ccc-46ef-895c-5ecc6ea4e599" (UID: "fae009e3-4ccc-46ef-895c-5ecc6ea4e599"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:34.474895 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.474421 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa8baac2-125d-42f6-8449-5fd590bf4b2b-kube-api-access-sb52g\" (UniqueName: \"kubernetes.io/projected/fa8baac2-125d-42f6-8449-5fd590bf4b2b-kube-api-access-sb52g\") pod \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\" (UID: \"fa8baac2-125d-42f6-8449-5fd590bf4b2b\") " Nov 03 22:08:34.474895 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.474505 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-tmp-dir\") pod \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\" (UID: \"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d\") " Nov 03 22:08:34.475244 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.475009 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.475244 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.475033 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.475244 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.475081 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.475244 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.475097 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.475244 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.475111 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.475526 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.475422 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-model-cache" pod "4524d0a6-727b-47a9-b3b9-e83e5cf3a52d" (UID: "4524d0a6-727b-47a9-b3b9-e83e5cf3a52d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:34.475667 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.475566 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-home" pod "fae009e3-4ccc-46ef-895c-5ecc6ea4e599" (UID: "fae009e3-4ccc-46ef-895c-5ecc6ea4e599"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:34.475756 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.475630 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-home" pod "4524d0a6-727b-47a9-b3b9-e83e5cf3a52d" (UID: "4524d0a6-727b-47a9-b3b9-e83e5cf3a52d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:34.475874 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.475845 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-tmp-dir" pod "4524d0a6-727b-47a9-b3b9-e83e5cf3a52d" (UID: "4524d0a6-727b-47a9-b3b9-e83e5cf3a52d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:34.476592 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.476560 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa8baac2-125d-42f6-8449-5fd590bf4b2b-tls-certs" pod "fa8baac2-125d-42f6-8449-5fd590bf4b2b" (UID: "fa8baac2-125d-42f6-8449-5fd590bf4b2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:34.476685 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.476612 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-dshm" pod "4524d0a6-727b-47a9-b3b9-e83e5cf3a52d" (UID: "4524d0a6-727b-47a9-b3b9-e83e5cf3a52d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:34.477503 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.477468 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-tls-certs" pod "4524d0a6-727b-47a9-b3b9-e83e5cf3a52d" (UID: "4524d0a6-727b-47a9-b3b9-e83e5cf3a52d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:34.477725 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.477695 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-dshm" pod "fae009e3-4ccc-46ef-895c-5ecc6ea4e599" (UID: "fae009e3-4ccc-46ef-895c-5ecc6ea4e599"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:34.478233 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.478208 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-kube-api-access-4jw9x" pod "4524d0a6-727b-47a9-b3b9-e83e5cf3a52d" (UID: "4524d0a6-727b-47a9-b3b9-e83e5cf3a52d"). InnerVolumeSpecName "kube-api-access-4jw9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:34.478316 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.478238 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-kube-api-access-npn2l" pod "fae009e3-4ccc-46ef-895c-5ecc6ea4e599" (UID: "fae009e3-4ccc-46ef-895c-5ecc6ea4e599"). InnerVolumeSpecName "kube-api-access-npn2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:34.478365 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.478313 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8baac2-125d-42f6-8449-5fd590bf4b2b-kube-api-access-sb52g" pod "fa8baac2-125d-42f6-8449-5fd590bf4b2b" (UID: "fa8baac2-125d-42f6-8449-5fd590bf4b2b"). InnerVolumeSpecName "kube-api-access-sb52g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:34.478474 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.478455 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-tls-certs" pod "fae009e3-4ccc-46ef-895c-5ecc6ea4e599" (UID: "fae009e3-4ccc-46ef-895c-5ecc6ea4e599"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:34.478558 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.478539 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-dshm" pod "fa8baac2-125d-42f6-8449-5fd590bf4b2b" (UID: "fa8baac2-125d-42f6-8449-5fd590bf4b2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:34.576407 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.576367 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.576407 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.576406 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.576681 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.576421 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.576681 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.576435 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa8baac2-125d-42f6-8449-5fd590bf4b2b-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.576681 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.576463 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npn2l\" (UniqueName: \"kubernetes.io/projected/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-kube-api-access-npn2l\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.576681 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.576487 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb52g\" (UniqueName: \"kubernetes.io/projected/fa8baac2-125d-42f6-8449-5fd590bf4b2b-kube-api-access-sb52g\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.576681 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.576522 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.576681 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.576536 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.576681 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.576548 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.576681 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.576561 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.576681 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.576570 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae009e3-4ccc-46ef-895c-5ecc6ea4e599-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.576681 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.576577 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa8baac2-125d-42f6-8449-5fd590bf4b2b-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:34.576681 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:34.576585 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jw9x\" (UniqueName: \"kubernetes.io/projected/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d-kube-api-access-4jw9x\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:35.010617 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.010553 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" podUID="d6b006c9-5373-4b54-b9a3-ae0ea5425005" containerName="main" containerID="cri-o://63ea8eb21ad652ca18d9dff5caef9d83b34ceef0700780c0e691413bdd9c5a91" gracePeriod=45 Nov 03 22:08:35.160937 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.160883 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" podUID="d6b006c9-5373-4b54-b9a3-ae0ea5425005" containerName="main" probeResult="failure" output="Get \"https://10.132.0.159:8000/ready\": dial tcp 10.132.0.159:8000: connect: connection refused" Nov 03 22:08:35.253062 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.253033 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:08:35.284239 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.284205 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6b006c9-5373-4b54-b9a3-ae0ea5425005-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b006c9-5373-4b54-b9a3-ae0ea5425005-tls-certs\") pod \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " Nov 03 22:08:35.284443 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.284248 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-model-cache\") pod \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " Nov 03 22:08:35.284443 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.284270 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-tmp-dir\") pod \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " Nov 03 22:08:35.284443 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.284384 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b006c9-5373-4b54-b9a3-ae0ea5425005-kube-api-access-gr4vl\" (UniqueName: \"kubernetes.io/projected/d6b006c9-5373-4b54-b9a3-ae0ea5425005-kube-api-access-gr4vl\") pod \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " Nov 03 22:08:35.284443 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.284410 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-home\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-home\") pod \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " Nov 03 22:08:35.284443 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.284433 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-dshm\") pod \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\" (UID: \"d6b006c9-5373-4b54-b9a3-ae0ea5425005\") " Nov 03 22:08:35.284715 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.284570 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-model-cache" pod "d6b006c9-5373-4b54-b9a3-ae0ea5425005" (UID: "d6b006c9-5373-4b54-b9a3-ae0ea5425005"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:35.284715 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.284591 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-tmp-dir" pod "d6b006c9-5373-4b54-b9a3-ae0ea5425005" (UID: "d6b006c9-5373-4b54-b9a3-ae0ea5425005"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:35.284715 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.284703 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-home" pod "d6b006c9-5373-4b54-b9a3-ae0ea5425005" (UID: "d6b006c9-5373-4b54-b9a3-ae0ea5425005"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:35.284854 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.284830 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:35.284854 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.284852 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:35.285014 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.284864 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:35.286559 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.286525 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b006c9-5373-4b54-b9a3-ae0ea5425005-tls-certs" pod "d6b006c9-5373-4b54-b9a3-ae0ea5425005" (UID: "d6b006c9-5373-4b54-b9a3-ae0ea5425005"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:35.286672 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.286575 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b006c9-5373-4b54-b9a3-ae0ea5425005-kube-api-access-gr4vl" pod "d6b006c9-5373-4b54-b9a3-ae0ea5425005" (UID: "d6b006c9-5373-4b54-b9a3-ae0ea5425005"). InnerVolumeSpecName "kube-api-access-gr4vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:35.286672 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.286590 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-dshm" pod "d6b006c9-5373-4b54-b9a3-ae0ea5425005" (UID: "d6b006c9-5373-4b54-b9a3-ae0ea5425005"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:35.334470 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.334425 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" Nov 03 22:08:35.334470 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.334449 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" event={"ID":"fa8baac2-125d-42f6-8449-5fd590bf4b2b","Type":"ContainerDied","Data":"d0fb90122402c6845d6b000e2bdbd1d091408617d37e2ac9119ed5ef9a3c098d"} Nov 03 22:08:35.334738 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.334511 4963 scope.go:122] "RemoveContainer" containerID="75d48a7aa0c9b43e9d4131786a80a1c965eb47dc57619b8399d71bed70b6c530" Nov 03 22:08:35.335981 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.335953 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" event={"ID":"fae009e3-4ccc-46ef-895c-5ecc6ea4e599","Type":"ContainerDied","Data":"9b70724036f76e090217737727c2f616ad3e69ea754db6104c01b8999555da71"} Nov 03 22:08:35.336092 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.335985 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f" Nov 03 22:08:35.337620 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.337592 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" Nov 03 22:08:35.337751 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.337627 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q" event={"ID":"4524d0a6-727b-47a9-b3b9-e83e5cf3a52d","Type":"ContainerDied","Data":"4e76a9a8c0a804303883ef897fb8c540b4748c439367ffb43668ca0c4cdbe243"} Nov 03 22:08:35.339564 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.339535 4963 generic.go:356] "Generic (PLEG): container finished" podID="d6b006c9-5373-4b54-b9a3-ae0ea5425005" containerID="63ea8eb21ad652ca18d9dff5caef9d83b34ceef0700780c0e691413bdd9c5a91" exitCode=0 Nov 03 22:08:35.339670 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.339573 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" Nov 03 22:08:35.339670 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.339581 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" event={"ID":"d6b006c9-5373-4b54-b9a3-ae0ea5425005","Type":"ContainerDied","Data":"63ea8eb21ad652ca18d9dff5caef9d83b34ceef0700780c0e691413bdd9c5a91"} Nov 03 22:08:35.339670 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.339612 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6" event={"ID":"d6b006c9-5373-4b54-b9a3-ae0ea5425005","Type":"ContainerDied","Data":"364ac6ac0ee0d7affbdd8351fef078c07ec1adc08d3809b42993647b4a7bf420"} Nov 03 22:08:35.346462 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.346433 4963 scope.go:122] "RemoveContainer" containerID="9186ab3aaac1096ee0dc3e212754aaef3d9b2c38a0ccec92bd791d20a973697a" Nov 03 22:08:35.355749 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.355726 4963 scope.go:122] "RemoveContainer" containerID="76861975ac0597c193c769913dfb0fec6dbf2f0e3fbfa1428d069c722139f015" Nov 03 22:08:35.364535 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.364514 4963 scope.go:122] "RemoveContainer" containerID="63ea8eb21ad652ca18d9dff5caef9d83b34ceef0700780c0e691413bdd9c5a91" Nov 03 22:08:35.382856 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.382007 4963 scope.go:122] "RemoveContainer" containerID="63ea8eb21ad652ca18d9dff5caef9d83b34ceef0700780c0e691413bdd9c5a91" Nov 03 22:08:35.382856 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:08:35.382511 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ea8eb21ad652ca18d9dff5caef9d83b34ceef0700780c0e691413bdd9c5a91\": container with ID starting with 63ea8eb21ad652ca18d9dff5caef9d83b34ceef0700780c0e691413bdd9c5a91 not found: ID does not exist" containerID="63ea8eb21ad652ca18d9dff5caef9d83b34ceef0700780c0e691413bdd9c5a91" Nov 03 22:08:35.382856 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.382554 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ea8eb21ad652ca18d9dff5caef9d83b34ceef0700780c0e691413bdd9c5a91"} err="failed to get container status \"63ea8eb21ad652ca18d9dff5caef9d83b34ceef0700780c0e691413bdd9c5a91\": rpc error: code = NotFound desc = could not find container \"63ea8eb21ad652ca18d9dff5caef9d83b34ceef0700780c0e691413bdd9c5a91\": container with ID starting with 63ea8eb21ad652ca18d9dff5caef9d83b34ceef0700780c0e691413bdd9c5a91 not found: ID does not exist" Nov 03 22:08:35.386411 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.386308 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr4vl\" (UniqueName: \"kubernetes.io/projected/d6b006c9-5373-4b54-b9a3-ae0ea5425005-kube-api-access-gr4vl\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:35.386411 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.386338 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b006c9-5373-4b54-b9a3-ae0ea5425005-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:35.386411 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.386353 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b006c9-5373-4b54-b9a3-ae0ea5425005-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:35.423561 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.423520 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x" podUID="fa8baac2-125d-42f6-8449-5fd590bf4b2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:08:35.471687 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.471642 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x"] Nov 03 22:08:35.476049 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.476016 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/premium-w2-099a5e-kserve-867dccb69-6zx5x"] Nov 03 22:08:35.580078 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.579988 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f"] Nov 03 22:08:35.584368 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.584335 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/distinct-2-w2-099a5e-kserve-5d7b88fc79-9nr9f"] Nov 03 22:08:35.688071 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.688034 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q"] Nov 03 22:08:35.691607 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.691577 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/distinct-w2-099a5e-kserve-6b5759886d-c7w4q"] Nov 03 22:08:35.796218 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.796184 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6"] Nov 03 22:08:35.800570 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.800542 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a1c634/sub-test-model-a1c634-kserve-6c5577c57-nhsd6"] Nov 03 22:08:35.837933 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:35.837818 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:36.967047 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:36.967013 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4524d0a6-727b-47a9-b3b9-e83e5cf3a52d" path="/var/lib/kubelet/pods/4524d0a6-727b-47a9-b3b9-e83e5cf3a52d/volumes" Nov 03 22:08:36.967466 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:36.967452 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b006c9-5373-4b54-b9a3-ae0ea5425005" path="/var/lib/kubelet/pods/d6b006c9-5373-4b54-b9a3-ae0ea5425005/volumes" Nov 03 22:08:36.967823 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:36.967810 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa8baac2-125d-42f6-8449-5fd590bf4b2b" path="/var/lib/kubelet/pods/fa8baac2-125d-42f6-8449-5fd590bf4b2b/volumes" Nov 03 22:08:36.968516 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:36.968502 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae009e3-4ccc-46ef-895c-5ecc6ea4e599" path="/var/lib/kubelet/pods/fae009e3-4ccc-46ef-895c-5ecc6ea4e599/volumes" Nov 03 22:08:37.909230 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:37.909192 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:38.094023 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:38.093988 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:38.373335 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:38.373306 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:39.102034 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:39.102000 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:39.284705 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:39.284672 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:40.602228 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:40.602195 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:40.778652 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:40.778620 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:41.079525 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:41.079495 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:41.831066 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:41.831024 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:42.527887 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:08:42.527849 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:08:44.221113 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:44.221076 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:44.852287 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:44.852252 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:45.032121 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:45.032090 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:45.425492 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:45.425459 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:48.489352 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:48.489306 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:48.688926 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:48.687783 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:48.867305 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:48.867210 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:49.614104 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:49.614062 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:51.395885 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:51.395824 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:54.309045 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:54.309000 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:55.423779 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:55.423742 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:56.805532 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:56.805490 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:56.987086 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:56.987049 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:57.173357 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:57.173264 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:08:57.387966 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:08:57.387939 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:00.639960 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:00.639896 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:00.827456 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:00.827410 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:01.202652 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:01.202613 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:01.393099 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:01.393066 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:01.707672 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:01.707629 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:02.138952 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:02.138795 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:02.712503 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:02.712470 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:03.415597 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:03.415537 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:03.887227 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:03.887192 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:04.220600 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:04.220568 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:04.420518 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:04.420476 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:06.807788 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:06.807743 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:09.276052 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:09.276013 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5"] Nov 03 22:09:09.323626 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:09.323587 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs"] Nov 03 22:09:09.325621 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:09.325591 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t"] Nov 03 22:09:09.415678 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:09.415646 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:11.432014 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:11.431737 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:12.527760 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:09:12.527719 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:09:15.530716 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:15.530682 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:16.151724 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:16.151691 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:20.749665 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:20.749627 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:21.767162 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:21.767132 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:22.429313 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:22.429273 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:22.611727 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:22.611695 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:24.298451 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.298409 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" podUID="8ce93966-0d0b-4b1d-aa96-16315a7467b5" containerName="main" containerID="cri-o://297be8dc97ece4b5b7f19d5ccea8ac0842f5531b629e0e338fdf4c03688d8fa6" gracePeriod=45 Nov 03 22:09:24.343660 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.343621 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" podUID="7d38fac8-2315-4693-8dae-4e449356ee4c" containerName="main" containerID="cri-o://c358e786cb0030bc7963220a79ec8f9e09f159d5bf56c24ffc7be69b5de5024d" gracePeriod=45 Nov 03 22:09:24.344898 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.344850 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" podUID="1edcbec2-0930-4b49-b14a-2d21568b5318" containerName="main" containerID="cri-o://41d64440d3501f8d1efb054654d3c8390c1694cece8a0ea5050aef97861367c5" gracePeriod=45 Nov 03 22:09:24.532745 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.532711 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" podUID="7d38fac8-2315-4693-8dae-4e449356ee4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": dial tcp 10.132.0.83:8000: connect: connection refused" Nov 03 22:09:24.607445 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.607405 4963 generic.go:356] "Generic (PLEG): container finished" podID="8ce93966-0d0b-4b1d-aa96-16315a7467b5" containerID="297be8dc97ece4b5b7f19d5ccea8ac0842f5531b629e0e338fdf4c03688d8fa6" exitCode=0 Nov 03 22:09:24.607655 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.607571 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" event={"ID":"8ce93966-0d0b-4b1d-aa96-16315a7467b5","Type":"ContainerDied","Data":"297be8dc97ece4b5b7f19d5ccea8ac0842f5531b629e0e338fdf4c03688d8fa6"} Nov 03 22:09:24.611307 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.610308 4963 generic.go:356] "Generic (PLEG): container finished" podID="7d38fac8-2315-4693-8dae-4e449356ee4c" containerID="c358e786cb0030bc7963220a79ec8f9e09f159d5bf56c24ffc7be69b5de5024d" exitCode=0 Nov 03 22:09:24.611307 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.610425 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" event={"ID":"7d38fac8-2315-4693-8dae-4e449356ee4c","Type":"ContainerDied","Data":"c358e786cb0030bc7963220a79ec8f9e09f159d5bf56c24ffc7be69b5de5024d"} Nov 03 22:09:24.613410 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.613380 4963 generic.go:356] "Generic (PLEG): container finished" podID="1edcbec2-0930-4b49-b14a-2d21568b5318" containerID="41d64440d3501f8d1efb054654d3c8390c1694cece8a0ea5050aef97861367c5" exitCode=0 Nov 03 22:09:24.613532 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.613419 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" event={"ID":"1edcbec2-0930-4b49-b14a-2d21568b5318","Type":"ContainerDied","Data":"41d64440d3501f8d1efb054654d3c8390c1694cece8a0ea5050aef97861367c5"} Nov 03 22:09:24.668088 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.668062 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 22:09:24.672097 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.672069 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 22:09:24.676643 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.676617 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 22:09:24.799609 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.799575 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-model-cache\") pod \"1edcbec2-0930-4b49-b14a-2d21568b5318\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " Nov 03 22:09:24.799801 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.799637 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-model-cache\") pod \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " Nov 03 22:09:24.799801 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.799671 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-dshm\") pod \"1edcbec2-0930-4b49-b14a-2d21568b5318\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " Nov 03 22:09:24.799801 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.799710 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-tmp-dir\") pod \"1edcbec2-0930-4b49-b14a-2d21568b5318\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " Nov 03 22:09:24.799801 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.799752 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-model-cache\") pod \"7d38fac8-2315-4693-8dae-4e449356ee4c\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " Nov 03 22:09:24.799801 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.799785 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-tmp-dir\") pod \"7d38fac8-2315-4693-8dae-4e449356ee4c\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " Nov 03 22:09:24.800129 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.799823 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1edcbec2-0930-4b49-b14a-2d21568b5318-kube-api-access-rsp25\" (UniqueName: \"kubernetes.io/projected/1edcbec2-0930-4b49-b14a-2d21568b5318-kube-api-access-rsp25\") pod \"1edcbec2-0930-4b49-b14a-2d21568b5318\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " Nov 03 22:09:24.800129 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.799884 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ce93966-0d0b-4b1d-aa96-16315a7467b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce93966-0d0b-4b1d-aa96-16315a7467b5-tls-certs\") pod \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " Nov 03 22:09:24.800129 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.799927 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-dshm\") pod \"7d38fac8-2315-4693-8dae-4e449356ee4c\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " Nov 03 22:09:24.800129 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.799932 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-model-cache" pod "1edcbec2-0930-4b49-b14a-2d21568b5318" (UID: "1edcbec2-0930-4b49-b14a-2d21568b5318"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:24.800129 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.799961 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce93966-0d0b-4b1d-aa96-16315a7467b5-kube-api-access-tr24z\" (UniqueName: \"kubernetes.io/projected/8ce93966-0d0b-4b1d-aa96-16315a7467b5-kube-api-access-tr24z\") pod \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " Nov 03 22:09:24.800129 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.799984 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-model-cache" pod "8ce93966-0d0b-4b1d-aa96-16315a7467b5" (UID: "8ce93966-0d0b-4b1d-aa96-16315a7467b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:24.800129 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.799995 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-home\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-home\") pod \"7d38fac8-2315-4693-8dae-4e449356ee4c\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " Nov 03 22:09:24.800129 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.800028 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1edcbec2-0930-4b49-b14a-2d21568b5318-tls-certs\" (UniqueName: \"kubernetes.io/secret/1edcbec2-0930-4b49-b14a-2d21568b5318-tls-certs\") pod \"1edcbec2-0930-4b49-b14a-2d21568b5318\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " Nov 03 22:09:24.800129 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.800079 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-dshm\") pod \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " Nov 03 22:09:24.800129 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.800129 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-home\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-home\") pod \"1edcbec2-0930-4b49-b14a-2d21568b5318\" (UID: \"1edcbec2-0930-4b49-b14a-2d21568b5318\") " Nov 03 22:09:24.800675 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.800174 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-home\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-home\") pod \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " Nov 03 22:09:24.800675 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.800217 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d38fac8-2315-4693-8dae-4e449356ee4c-kube-api-access-vt8hk\" (UniqueName: \"kubernetes.io/projected/7d38fac8-2315-4693-8dae-4e449356ee4c-kube-api-access-vt8hk\") pod \"7d38fac8-2315-4693-8dae-4e449356ee4c\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " Nov 03 22:09:24.800675 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.800243 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-tmp-dir\") pod \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\" (UID: \"8ce93966-0d0b-4b1d-aa96-16315a7467b5\") " Nov 03 22:09:24.800675 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.800278 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d38fac8-2315-4693-8dae-4e449356ee4c-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d38fac8-2315-4693-8dae-4e449356ee4c-tls-certs\") pod \"7d38fac8-2315-4693-8dae-4e449356ee4c\" (UID: \"7d38fac8-2315-4693-8dae-4e449356ee4c\") " Nov 03 22:09:24.800675 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.800473 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-tmp-dir" pod "7d38fac8-2315-4693-8dae-4e449356ee4c" (UID: "7d38fac8-2315-4693-8dae-4e449356ee4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:24.800675 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.800512 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-home" pod "1edcbec2-0930-4b49-b14a-2d21568b5318" (UID: "1edcbec2-0930-4b49-b14a-2d21568b5318"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:24.800675 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.800598 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-model-cache" pod "7d38fac8-2315-4693-8dae-4e449356ee4c" (UID: "7d38fac8-2315-4693-8dae-4e449356ee4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:24.801068 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.800707 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:24.801068 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.800727 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:24.801068 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.800743 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:24.801068 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.800757 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:24.801068 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.800772 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:24.801068 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.801042 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-home" pod "7d38fac8-2315-4693-8dae-4e449356ee4c" (UID: "7d38fac8-2315-4693-8dae-4e449356ee4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:24.801940 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.801247 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-home" pod "8ce93966-0d0b-4b1d-aa96-16315a7467b5" (UID: "8ce93966-0d0b-4b1d-aa96-16315a7467b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:24.801940 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.801268 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-tmp-dir" pod "8ce93966-0d0b-4b1d-aa96-16315a7467b5" (UID: "8ce93966-0d0b-4b1d-aa96-16315a7467b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:24.801940 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.801306 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-tmp-dir" pod "1edcbec2-0930-4b49-b14a-2d21568b5318" (UID: "1edcbec2-0930-4b49-b14a-2d21568b5318"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:24.803821 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.803764 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1edcbec2-0930-4b49-b14a-2d21568b5318-tls-certs" pod "1edcbec2-0930-4b49-b14a-2d21568b5318" (UID: "1edcbec2-0930-4b49-b14a-2d21568b5318"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:09:24.803965 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.803841 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d38fac8-2315-4693-8dae-4e449356ee4c-tls-certs" pod "7d38fac8-2315-4693-8dae-4e449356ee4c" (UID: "7d38fac8-2315-4693-8dae-4e449356ee4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:09:24.804174 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.804133 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-dshm" pod "1edcbec2-0930-4b49-b14a-2d21568b5318" (UID: "1edcbec2-0930-4b49-b14a-2d21568b5318"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:24.804174 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.804149 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edcbec2-0930-4b49-b14a-2d21568b5318-kube-api-access-rsp25" pod "1edcbec2-0930-4b49-b14a-2d21568b5318" (UID: "1edcbec2-0930-4b49-b14a-2d21568b5318"). InnerVolumeSpecName "kube-api-access-rsp25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:24.804378 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.804169 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ce93966-0d0b-4b1d-aa96-16315a7467b5-tls-certs" pod "8ce93966-0d0b-4b1d-aa96-16315a7467b5" (UID: "8ce93966-0d0b-4b1d-aa96-16315a7467b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:09:24.804536 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.804512 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-dshm" pod "7d38fac8-2315-4693-8dae-4e449356ee4c" (UID: "7d38fac8-2315-4693-8dae-4e449356ee4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:24.804995 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.804965 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce93966-0d0b-4b1d-aa96-16315a7467b5-kube-api-access-tr24z" pod "8ce93966-0d0b-4b1d-aa96-16315a7467b5" (UID: "8ce93966-0d0b-4b1d-aa96-16315a7467b5"). InnerVolumeSpecName "kube-api-access-tr24z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:24.805523 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.805504 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-dshm" pod "8ce93966-0d0b-4b1d-aa96-16315a7467b5" (UID: "8ce93966-0d0b-4b1d-aa96-16315a7467b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:24.805647 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.805630 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d38fac8-2315-4693-8dae-4e449356ee4c-kube-api-access-vt8hk" pod "7d38fac8-2315-4693-8dae-4e449356ee4c" (UID: "7d38fac8-2315-4693-8dae-4e449356ee4c"). InnerVolumeSpecName "kube-api-access-vt8hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:24.902268 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.902190 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:24.902268 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.902220 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1edcbec2-0930-4b49-b14a-2d21568b5318-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:24.902268 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.902234 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsp25\" (UniqueName: \"kubernetes.io/projected/1edcbec2-0930-4b49-b14a-2d21568b5318-kube-api-access-rsp25\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:24.902268 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.902248 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce93966-0d0b-4b1d-aa96-16315a7467b5-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:24.902268 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.902261 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:24.902571 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.902274 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr24z\" (UniqueName: \"kubernetes.io/projected/8ce93966-0d0b-4b1d-aa96-16315a7467b5-kube-api-access-tr24z\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:24.902571 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.902286 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d38fac8-2315-4693-8dae-4e449356ee4c-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:24.902571 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.902297 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1edcbec2-0930-4b49-b14a-2d21568b5318-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:24.902571 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.902307 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:24.902571 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.902318 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:24.902571 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.902329 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt8hk\" (UniqueName: \"kubernetes.io/projected/7d38fac8-2315-4693-8dae-4e449356ee4c-kube-api-access-vt8hk\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:24.902571 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.902343 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce93966-0d0b-4b1d-aa96-16315a7467b5-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:24.902571 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:24.902354 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d38fac8-2315-4693-8dae-4e449356ee4c-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:25.620036 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:25.619947 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" Nov 03 22:09:25.620487 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:25.619932 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs" event={"ID":"7d38fac8-2315-4693-8dae-4e449356ee4c","Type":"ContainerDied","Data":"50a0eab55d034dae100e133df62790f4e4a80b80b9b534f01185f75a39ec5406"} Nov 03 22:09:25.620487 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:25.620125 4963 scope.go:122] "RemoveContainer" containerID="c358e786cb0030bc7963220a79ec8f9e09f159d5bf56c24ffc7be69b5de5024d" Nov 03 22:09:25.621616 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:25.621516 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" event={"ID":"1edcbec2-0930-4b49-b14a-2d21568b5318","Type":"ContainerDied","Data":"0658bf15bd1dd2ab8141920f3f78c9f190e3063a8a54c9dbec9ef0e850dce0ca"} Nov 03 22:09:25.621616 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:25.621532 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t" Nov 03 22:09:25.623132 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:25.623111 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" Nov 03 22:09:25.623243 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:25.623108 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5" event={"ID":"8ce93966-0d0b-4b1d-aa96-16315a7467b5","Type":"ContainerDied","Data":"cfd0ecb19712e6dfdddf208a1cf948315c3c09a176be5721d7189ac4b7f146ce"} Nov 03 22:09:25.630514 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:25.630489 4963 scope.go:122] "RemoveContainer" containerID="41d64440d3501f8d1efb054654d3c8390c1694cece8a0ea5050aef97861367c5" Nov 03 22:09:25.644505 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:25.644478 4963 scope.go:122] "RemoveContainer" containerID="297be8dc97ece4b5b7f19d5ccea8ac0842f5531b629e0e338fdf4c03688d8fa6" Nov 03 22:09:25.783978 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:25.783945 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t"] Nov 03 22:09:25.831492 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:25.831459 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/simulator-w4-e84357-kserve-5fd8c78999-zgv4t"] Nov 03 22:09:25.897108 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:25.897017 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs"] Nov 03 22:09:25.900360 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:25.900331 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/distinct-w4-e84357-kserve-58bd574449-8g8gs"] Nov 03 22:09:26.005180 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:26.005143 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5"] Nov 03 22:09:26.019561 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:26.019520 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/distinct-2-w4-e84357-kserve-88f87ddf4-xgnm5"] Nov 03 22:09:26.966674 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:26.966631 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1edcbec2-0930-4b49-b14a-2d21568b5318" path="/var/lib/kubelet/pods/1edcbec2-0930-4b49-b14a-2d21568b5318/volumes" Nov 03 22:09:26.967165 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:26.967148 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d38fac8-2315-4693-8dae-4e449356ee4c" path="/var/lib/kubelet/pods/7d38fac8-2315-4693-8dae-4e449356ee4c/volumes" Nov 03 22:09:26.967566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:26.967553 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ce93966-0d0b-4b1d-aa96-16315a7467b5" path="/var/lib/kubelet/pods/8ce93966-0d0b-4b1d-aa96-16315a7467b5/volumes" Nov 03 22:09:28.666784 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:28.666742 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:34.411372 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:34.411335 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:34.929622 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:34.929583 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph"] Nov 03 22:09:34.929858 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:34.929836 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" podUID="679af1f2-dd8a-4af5-a8d8-518a4a0db304" containerName="maas-api" containerID="cri-o://f9b9517eaf0e448bc6c7053160386381e58cfa4a585aefce648f50ead8286256" gracePeriod=30 Nov 03 22:09:35.073625 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.073591 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:35.187893 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.187820 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 22:09:35.210840 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.210800 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/679af1f2-dd8a-4af5-a8d8-518a4a0db304-kube-api-access-86b9b\" (UniqueName: \"kubernetes.io/projected/679af1f2-dd8a-4af5-a8d8-518a4a0db304-kube-api-access-86b9b\") pod \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\" (UID: \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\") " Nov 03 22:09:35.210840 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.210847 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/679af1f2-dd8a-4af5-a8d8-518a4a0db304-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679af1f2-dd8a-4af5-a8d8-518a4a0db304-maas-api-tls\") pod \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\" (UID: \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\") " Nov 03 22:09:35.211138 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.210878 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/679af1f2-dd8a-4af5-a8d8-518a4a0db304-metrics-certs\" (UniqueName: \"kubernetes.io/secret/679af1f2-dd8a-4af5-a8d8-518a4a0db304-metrics-certs\") pod \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\" (UID: \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\") " Nov 03 22:09:35.211138 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.210977 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/679af1f2-dd8a-4af5-a8d8-518a4a0db304-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/679af1f2-dd8a-4af5-a8d8-518a4a0db304-openshift-service-ca\") pod \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\" (UID: \"679af1f2-dd8a-4af5-a8d8-518a4a0db304\") " Nov 03 22:09:35.211371 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.211335 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679af1f2-dd8a-4af5-a8d8-518a4a0db304-openshift-service-ca" pod "679af1f2-dd8a-4af5-a8d8-518a4a0db304" (UID: "679af1f2-dd8a-4af5-a8d8-518a4a0db304"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:09:35.213153 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.213121 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/679af1f2-dd8a-4af5-a8d8-518a4a0db304-maas-api-tls" pod "679af1f2-dd8a-4af5-a8d8-518a4a0db304" (UID: "679af1f2-dd8a-4af5-a8d8-518a4a0db304"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:09:35.213259 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.213188 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/679af1f2-dd8a-4af5-a8d8-518a4a0db304-metrics-certs" pod "679af1f2-dd8a-4af5-a8d8-518a4a0db304" (UID: "679af1f2-dd8a-4af5-a8d8-518a4a0db304"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:09:35.213259 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.213196 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679af1f2-dd8a-4af5-a8d8-518a4a0db304-kube-api-access-86b9b" pod "679af1f2-dd8a-4af5-a8d8-518a4a0db304" (UID: "679af1f2-dd8a-4af5-a8d8-518a4a0db304"). InnerVolumeSpecName "kube-api-access-86b9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:35.311786 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.311746 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/679af1f2-dd8a-4af5-a8d8-518a4a0db304-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:35.311786 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.311787 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/679af1f2-dd8a-4af5-a8d8-518a4a0db304-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:35.312006 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.311804 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86b9b\" (UniqueName: \"kubernetes.io/projected/679af1f2-dd8a-4af5-a8d8-518a4a0db304-kube-api-access-86b9b\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:35.312006 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.311819 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679af1f2-dd8a-4af5-a8d8-518a4a0db304-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:35.677167 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.677131 4963 generic.go:356] "Generic (PLEG): container finished" podID="679af1f2-dd8a-4af5-a8d8-518a4a0db304" containerID="f9b9517eaf0e448bc6c7053160386381e58cfa4a585aefce648f50ead8286256" exitCode=0 Nov 03 22:09:35.677167 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.677172 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" event={"ID":"679af1f2-dd8a-4af5-a8d8-518a4a0db304","Type":"ContainerDied","Data":"f9b9517eaf0e448bc6c7053160386381e58cfa4a585aefce648f50ead8286256"} Nov 03 22:09:35.677677 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.677175 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" Nov 03 22:09:35.677677 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.677219 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph" event={"ID":"679af1f2-dd8a-4af5-a8d8-518a4a0db304","Type":"ContainerDied","Data":"dbc5fd2d189f183270cd4a1d5f99bbaaf000018b0c950b1ffc5f0d7509051a83"} Nov 03 22:09:35.677677 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.677254 4963 scope.go:122] "RemoveContainer" containerID="f9b9517eaf0e448bc6c7053160386381e58cfa4a585aefce648f50ead8286256" Nov 03 22:09:35.687666 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.687643 4963 scope.go:122] "RemoveContainer" containerID="f9b9517eaf0e448bc6c7053160386381e58cfa4a585aefce648f50ead8286256" Nov 03 22:09:35.687984 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:09:35.687965 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b9517eaf0e448bc6c7053160386381e58cfa4a585aefce648f50ead8286256\": container with ID starting with f9b9517eaf0e448bc6c7053160386381e58cfa4a585aefce648f50ead8286256 not found: ID does not exist" containerID="f9b9517eaf0e448bc6c7053160386381e58cfa4a585aefce648f50ead8286256" Nov 03 22:09:35.688067 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.687991 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b9517eaf0e448bc6c7053160386381e58cfa4a585aefce648f50ead8286256"} err="failed to get container status \"f9b9517eaf0e448bc6c7053160386381e58cfa4a585aefce648f50ead8286256\": rpc error: code = NotFound desc = could not find container \"f9b9517eaf0e448bc6c7053160386381e58cfa4a585aefce648f50ead8286256\": container with ID starting with f9b9517eaf0e448bc6c7053160386381e58cfa4a585aefce648f50ead8286256 not found: ID does not exist" Nov 03 22:09:35.833910 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.833862 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph"] Nov 03 22:09:35.881334 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:35.881297 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e84357-6546b97b67-lz7ph"] Nov 03 22:09:36.965969 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:36.965934 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="679af1f2-dd8a-4af5-a8d8-518a4a0db304" path="/var/lib/kubelet/pods/679af1f2-dd8a-4af5-a8d8-518a4a0db304/volumes" Nov 03 22:09:39.992957 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:39.992894 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:40.302188 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:40.302102 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:42.527925 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:09:42.527878 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:09:46.763136 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:46.763101 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:46.946695 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:46.946663 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:47.133236 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:47.133156 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:48.861567 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:48.861532 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:50.217244 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:50.217195 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:50.640807 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:50.640729 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9"] Nov 03 22:09:50.641098 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:50.640950 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" podUID="cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" containerName="istio-proxy" containerID="cri-o://ef1ec71e298fee1c3a71e831a4c80c2498cc1a16d0db94f20615b0e08874a9b5" gracePeriod=30 Nov 03 22:09:50.686654 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:50.686615 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:50.873180 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:50.873145 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:51.648232 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:51.648199 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-579685d67f-v4cx7_7e175791-e600-4f34-b7f4-688619a990e1/authorino/0.log" Nov 03 22:09:52.042446 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:52.042414 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-mcs6w_3a26aaaf-9eac-483f-b59e-2fdd595e214c/manager/0.log" Nov 03 22:09:52.291815 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:52.291787 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lcfpk_5359b257-aafe-4a02-b987-786f2b8e3efc/manager/0.log" Nov 03 22:09:52.801780 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:52.801746 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:52.989399 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:52.989363 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:09:54.808011 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:54.807969 4963 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:09:54.808408 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:54.808025 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" podUID="cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:09:55.790033 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:55.789997 4963 generic.go:356] "Generic (PLEG): container finished" podID="cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" containerID="ef1ec71e298fee1c3a71e831a4c80c2498cc1a16d0db94f20615b0e08874a9b5" exitCode=0 Nov 03 22:09:55.790231 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:55.790094 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" event={"ID":"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d","Type":"ContainerDied","Data":"ef1ec71e298fee1c3a71e831a4c80c2498cc1a16d0db94f20615b0e08874a9b5"} Nov 03 22:09:55.896597 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:55.896568 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 22:09:56.010590 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.010559 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-podinfo\") pod \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " Nov 03 22:09:56.010590 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.010601 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-workload-socket\") pod \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " Nov 03 22:09:56.010842 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.010626 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-token\" (UniqueName: \"kubernetes.io/projected/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-token\") pod \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " Nov 03 22:09:56.010842 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.010668 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istiod-ca-cert\") pod \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " Nov 03 22:09:56.010842 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.010692 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-envoy\") pod \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " Nov 03 22:09:56.010842 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.010716 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-data\") pod \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " Nov 03 22:09:56.010842 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.010815 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-credential-socket\") pod \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " Nov 03 22:09:56.011197 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.010863 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-workload-certs\") pod \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " Nov 03 22:09:56.011197 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.010897 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-kube-api-access-68n6k\" (UniqueName: \"kubernetes.io/projected/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-kube-api-access-68n6k\") pod \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\" (UID: \"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d\") " Nov 03 22:09:56.011197 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.010931 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-workload-socket" pod "cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" (UID: "cfe1449b-31f2-45a8-b054-d4dc8ec7e72d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:56.011197 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.011187 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-credential-socket" pod "cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" (UID: "cfe1449b-31f2-45a8-b054-d4dc8ec7e72d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:56.011430 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.011202 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istiod-ca-cert" pod "cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" (UID: "cfe1449b-31f2-45a8-b054-d4dc8ec7e72d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:09:56.011430 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.011208 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-workload-certs" pod "cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" (UID: "cfe1449b-31f2-45a8-b054-d4dc8ec7e72d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:56.011430 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.011322 4963 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-workload-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:56.011430 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.011342 4963 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istiod-ca-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:56.011430 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.011357 4963 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-credential-socket\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:56.011430 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.011332 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-data" pod "cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" (UID: "cfe1449b-31f2-45a8-b054-d4dc8ec7e72d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:56.013248 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.013224 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-kube-api-access-68n6k" pod "cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" (UID: "cfe1449b-31f2-45a8-b054-d4dc8ec7e72d"). InnerVolumeSpecName "kube-api-access-68n6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:56.013248 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.013235 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-envoy" pod "cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" (UID: "cfe1449b-31f2-45a8-b054-d4dc8ec7e72d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:09:56.013570 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.013463 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-podinfo" pod "cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" (UID: "cfe1449b-31f2-45a8-b054-d4dc8ec7e72d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:09:56.013570 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.013497 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-token" pod "cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" (UID: "cfe1449b-31f2-45a8-b054-d4dc8ec7e72d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:09:56.112165 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.112131 4963 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-workload-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:56.112165 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.112162 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68n6k\" (UniqueName: \"kubernetes.io/projected/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-kube-api-access-68n6k\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:56.112165 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.112174 4963 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-podinfo\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:56.112423 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.112184 4963 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-token\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:56.112423 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.112193 4963 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-envoy\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:56.112423 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.112201 4963 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d-istio-data\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:09:56.507657 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.507594 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6m794_24076c9b-c504-4b0c-82dc-d95b44f0e977/manager/0.log" Nov 03 22:09:56.796615 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.796527 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" Nov 03 22:09:56.796615 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.796538 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9" event={"ID":"cfe1449b-31f2-45a8-b054-d4dc8ec7e72d","Type":"ContainerDied","Data":"48f1f5f08511a422c408f3219d0505efec58defdb490f71d082a0517f579519f"} Nov 03 22:09:56.796863 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.796622 4963 scope.go:122] "RemoveContainer" containerID="ef1ec71e298fee1c3a71e831a4c80c2498cc1a16d0db94f20615b0e08874a9b5" Nov 03 22:09:56.973401 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:56.973342 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77b845648-gxp5x_e5fac629-6443-4e4e-a5b3-08a949199c32/manager/0.log" Nov 03 22:09:57.002338 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:57.002303 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9"] Nov 03 22:09:57.049728 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:57.049636 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-4w7n9"] Nov 03 22:09:57.471148 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:57.471115 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-jww6w_fd4d8972-0ca6-40d6-8bd7-993d611500b4/manager/0.log" Nov 03 22:09:57.740753 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:57.740670 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-gdzc5_760f65bf-694d-407a-888b-c408028109f6/proxy/0.log" Nov 03 22:09:58.966037 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:09:58.965998 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" path="/var/lib/kubelet/pods/cfe1449b-31f2-45a8-b054-d4dc8ec7e72d/volumes" Nov 03 22:10:02.664302 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:02.664259 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:03.301672 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:03.301622 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:03.488463 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:03.488427 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:06.436692 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:06.436658 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:06.633267 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:06.633228 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:08.503333 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:08.503297 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:09.694026 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:09.693996 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:09.883002 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:09.882956 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:12.288953 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:12.288922 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:12.479223 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:12.479195 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:12.526340 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:10:12.526302 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:10:12.664387 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:12.664290 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:16.902512 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:16.902455 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:19.961455 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:19.961394 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:21.513270 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:21.513228 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:26.023291 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:26.023258 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:30.814286 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:30.814248 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:35.689286 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:35.689246 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:38.653208 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:38.653137 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:40.408972 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:40.408859 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:42.527825 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:10:42.527784 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:10:46.648801 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:46.648764 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:49.204506 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:49.204465 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:49.510531 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:49.510432 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:54.019339 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:54.019295 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:54.747419 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:54.747385 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:54.945683 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:54.945651 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:58.014032 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:58.013988 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:59.442544 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:59.442507 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:10:59.709069 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:10:59.708960 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:04.025217 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:04.025164 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:08.452098 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:08.452057 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77b845648-gxp5x"] Nov 03 22:11:08.452573 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:08.452235 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-77b845648-gxp5x" podUID="e5fac629-6443-4e4e-a5b3-08a949199c32" containerName="manager" containerID="cri-o://2a8fbd5a72201e13aa47440a034cc7166037b735ef374235b1427077992f3629" gracePeriod=30 Nov 03 22:11:08.702641 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:08.702570 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77b845648-gxp5x" Nov 03 22:11:08.791707 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:08.791670 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5fac629-6443-4e4e-a5b3-08a949199c32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5fac629-6443-4e4e-a5b3-08a949199c32-metrics-certs\") pod \"e5fac629-6443-4e4e-a5b3-08a949199c32\" (UID: \"e5fac629-6443-4e4e-a5b3-08a949199c32\") " Nov 03 22:11:08.791707 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:08.791718 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5fac629-6443-4e4e-a5b3-08a949199c32-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e5fac629-6443-4e4e-a5b3-08a949199c32-webhook-cert\") pod \"e5fac629-6443-4e4e-a5b3-08a949199c32\" (UID: \"e5fac629-6443-4e4e-a5b3-08a949199c32\") " Nov 03 22:11:08.792018 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:08.791742 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5fac629-6443-4e4e-a5b3-08a949199c32-kube-api-access-p2p72\" (UniqueName: \"kubernetes.io/projected/e5fac629-6443-4e4e-a5b3-08a949199c32-kube-api-access-p2p72\") pod \"e5fac629-6443-4e4e-a5b3-08a949199c32\" (UID: \"e5fac629-6443-4e4e-a5b3-08a949199c32\") " Nov 03 22:11:08.793953 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:08.793916 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5fac629-6443-4e4e-a5b3-08a949199c32-metrics-certs" pod "e5fac629-6443-4e4e-a5b3-08a949199c32" (UID: "e5fac629-6443-4e4e-a5b3-08a949199c32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:11:08.794103 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:08.794038 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5fac629-6443-4e4e-a5b3-08a949199c32-webhook-cert" pod "e5fac629-6443-4e4e-a5b3-08a949199c32" (UID: "e5fac629-6443-4e4e-a5b3-08a949199c32"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:11:08.794103 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:08.794052 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fac629-6443-4e4e-a5b3-08a949199c32-kube-api-access-p2p72" pod "e5fac629-6443-4e4e-a5b3-08a949199c32" (UID: "e5fac629-6443-4e4e-a5b3-08a949199c32"). InnerVolumeSpecName "kube-api-access-p2p72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:11:08.893206 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:08.893163 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5fac629-6443-4e4e-a5b3-08a949199c32-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:11:08.893206 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:08.893204 4963 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e5fac629-6443-4e4e-a5b3-08a949199c32-webhook-cert\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:11:08.893425 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:08.893223 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2p72\" (UniqueName: \"kubernetes.io/projected/e5fac629-6443-4e4e-a5b3-08a949199c32-kube-api-access-p2p72\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:11:09.155561 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:09.155520 4963 generic.go:356] "Generic (PLEG): container finished" podID="e5fac629-6443-4e4e-a5b3-08a949199c32" containerID="2a8fbd5a72201e13aa47440a034cc7166037b735ef374235b1427077992f3629" exitCode=0 Nov 03 22:11:09.155741 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:09.155571 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77b845648-gxp5x" event={"ID":"e5fac629-6443-4e4e-a5b3-08a949199c32","Type":"ContainerDied","Data":"2a8fbd5a72201e13aa47440a034cc7166037b735ef374235b1427077992f3629"} Nov 03 22:11:09.155741 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:09.155612 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77b845648-gxp5x" event={"ID":"e5fac629-6443-4e4e-a5b3-08a949199c32","Type":"ContainerDied","Data":"a3940ce6df687c77217555c4a73dc608a32a83a92511626c71b2acd96b125f4f"} Nov 03 22:11:09.155741 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:09.155636 4963 scope.go:122] "RemoveContainer" containerID="2a8fbd5a72201e13aa47440a034cc7166037b735ef374235b1427077992f3629" Nov 03 22:11:09.155741 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:09.155577 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77b845648-gxp5x" Nov 03 22:11:09.166186 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:09.166165 4963 scope.go:122] "RemoveContainer" containerID="2a8fbd5a72201e13aa47440a034cc7166037b735ef374235b1427077992f3629" Nov 03 22:11:09.166483 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:11:09.166460 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a8fbd5a72201e13aa47440a034cc7166037b735ef374235b1427077992f3629\": container with ID starting with 2a8fbd5a72201e13aa47440a034cc7166037b735ef374235b1427077992f3629 not found: ID does not exist" containerID="2a8fbd5a72201e13aa47440a034cc7166037b735ef374235b1427077992f3629" Nov 03 22:11:09.166564 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:09.166498 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8fbd5a72201e13aa47440a034cc7166037b735ef374235b1427077992f3629"} err="failed to get container status \"2a8fbd5a72201e13aa47440a034cc7166037b735ef374235b1427077992f3629\": rpc error: code = NotFound desc = could not find container \"2a8fbd5a72201e13aa47440a034cc7166037b735ef374235b1427077992f3629\": container with ID starting with 2a8fbd5a72201e13aa47440a034cc7166037b735ef374235b1427077992f3629 not found: ID does not exist" Nov 03 22:11:09.312971 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:09.312850 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77b845648-gxp5x"] Nov 03 22:11:09.317734 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:09.317681 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77b845648-gxp5x"] Nov 03 22:11:10.966656 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:10.966613 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5fac629-6443-4e4e-a5b3-08a949199c32" path="/var/lib/kubelet/pods/e5fac629-6443-4e4e-a5b3-08a949199c32/volumes" Nov 03 22:11:12.529455 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:11:12.529409 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:11:42.527397 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:11:42.527358 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:11:43.133889 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.133855 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8"] Nov 03 22:11:43.134269 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134255 4963 cpu_manager.go:414] "containerMap: removing container" podUID="e5fac629-6443-4e4e-a5b3-08a949199c32" containerName="manager" Nov 03 22:11:43.134320 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134270 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5fac629-6443-4e4e-a5b3-08a949199c32" containerName="manager" Nov 03 22:11:43.134320 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134282 4963 cpu_manager.go:414] "containerMap: removing container" podUID="0705edee-7b08-4a0d-b47c-7b8ce7f3279a" containerName="maas-api" Nov 03 22:11:43.134320 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134287 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0705edee-7b08-4a0d-b47c-7b8ce7f3279a" containerName="maas-api" Nov 03 22:11:43.134320 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134298 4963 cpu_manager.go:414] "containerMap: removing container" podUID="b48999ea-6a3b-427c-a22b-621a5cc997db" containerName="main" Nov 03 22:11:43.134320 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134303 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48999ea-6a3b-427c-a22b-621a5cc997db" containerName="main" Nov 03 22:11:43.134320 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134310 4963 cpu_manager.go:414] "containerMap: removing container" podUID="8ce93966-0d0b-4b1d-aa96-16315a7467b5" containerName="main" Nov 03 22:11:43.134320 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134314 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ce93966-0d0b-4b1d-aa96-16315a7467b5" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134325 4963 cpu_manager.go:414] "containerMap: removing container" podUID="58309a66-dfdf-4793-bbc5-5c7b33ca111e" containerName="payload-processing" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134330 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58309a66-dfdf-4793-bbc5-5c7b33ca111e" containerName="payload-processing" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134336 4963 cpu_manager.go:414] "containerMap: removing container" podUID="cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" containerName="istio-proxy" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134341 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" containerName="istio-proxy" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134348 4963 cpu_manager.go:414] "containerMap: removing container" podUID="62344e41-1f57-4ca4-9805-668cec58dd3b" containerName="maas-api" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134354 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62344e41-1f57-4ca4-9805-668cec58dd3b" containerName="maas-api" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134360 4963 cpu_manager.go:414] "containerMap: removing container" podUID="4524d0a6-727b-47a9-b3b9-e83e5cf3a52d" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134364 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4524d0a6-727b-47a9-b3b9-e83e5cf3a52d" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134372 4963 cpu_manager.go:414] "containerMap: removing container" podUID="fae009e3-4ccc-46ef-895c-5ecc6ea4e599" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134377 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae009e3-4ccc-46ef-895c-5ecc6ea4e599" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134389 4963 cpu_manager.go:414] "containerMap: removing container" podUID="cf9455eb-457d-4fec-aca7-f55202209d2c" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134393 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9455eb-457d-4fec-aca7-f55202209d2c" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134400 4963 cpu_manager.go:414] "containerMap: removing container" podUID="7eef8d7a-0034-40d0-9027-d6c4528dd24e" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134404 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eef8d7a-0034-40d0-9027-d6c4528dd24e" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134410 4963 cpu_manager.go:414] "containerMap: removing container" podUID="7d38fac8-2315-4693-8dae-4e449356ee4c" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134415 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d38fac8-2315-4693-8dae-4e449356ee4c" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134421 4963 cpu_manager.go:414] "containerMap: removing container" podUID="4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a" containerName="maas-api" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134425 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a" containerName="maas-api" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134431 4963 cpu_manager.go:414] "containerMap: removing container" podUID="98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" containerName="istio-proxy" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134436 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" containerName="istio-proxy" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134451 4963 cpu_manager.go:414] "containerMap: removing container" podUID="9204dab2-ccbe-4405-b9c7-666f23309bbd" containerName="payload-pre-processing" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134457 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9204dab2-ccbe-4405-b9c7-666f23309bbd" containerName="payload-pre-processing" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134468 4963 cpu_manager.go:414] "containerMap: removing container" podUID="7bdfcd76-3dd3-46fd-a382-0e40e648aa17" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134474 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bdfcd76-3dd3-46fd-a382-0e40e648aa17" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134483 4963 cpu_manager.go:414] "containerMap: removing container" podUID="e33113d6-3645-47a5-8f09-edee50ffb54d" containerName="maas-api" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134490 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e33113d6-3645-47a5-8f09-edee50ffb54d" containerName="maas-api" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134508 4963 cpu_manager.go:414] "containerMap: removing container" podUID="cb3fc56a-e7a0-445a-b206-f903fd3d12d9" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134515 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3fc56a-e7a0-445a-b206-f903fd3d12d9" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134528 4963 cpu_manager.go:414] "containerMap: removing container" podUID="1edcbec2-0930-4b49-b14a-2d21568b5318" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134535 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1edcbec2-0930-4b49-b14a-2d21568b5318" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134543 4963 cpu_manager.go:414] "containerMap: removing container" podUID="662bceeb-beae-4a4f-b08a-08706bed2d2e" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134550 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="662bceeb-beae-4a4f-b08a-08706bed2d2e" containerName="main" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134563 4963 cpu_manager.go:414] "containerMap: removing container" podUID="eda7b470-9598-458c-a270-4b0789924f39" containerName="istio-proxy" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134569 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eda7b470-9598-458c-a270-4b0789924f39" containerName="istio-proxy" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134579 4963 cpu_manager.go:414] "containerMap: removing container" podUID="70e4f9f6-f644-40b2-93fb-0acb72d86443" containerName="istio-proxy" Nov 03 22:11:43.134566 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134584 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e4f9f6-f644-40b2-93fb-0acb72d86443" containerName="istio-proxy" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134596 4963 cpu_manager.go:414] "containerMap: removing container" podUID="679af1f2-dd8a-4af5-a8d8-518a4a0db304" containerName="maas-api" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134600 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679af1f2-dd8a-4af5-a8d8-518a4a0db304" containerName="maas-api" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134607 4963 cpu_manager.go:414] "containerMap: removing container" podUID="fa8baac2-125d-42f6-8449-5fd590bf4b2b" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134612 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8baac2-125d-42f6-8449-5fd590bf4b2b" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134619 4963 cpu_manager.go:414] "containerMap: removing container" podUID="a3f5464f-5d94-499e-9fee-7553aa1e78a4" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134623 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f5464f-5d94-499e-9fee-7553aa1e78a4" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134628 4963 cpu_manager.go:414] "containerMap: removing container" podUID="d6b006c9-5373-4b54-b9a3-ae0ea5425005" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134633 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b006c9-5373-4b54-b9a3-ae0ea5425005" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134720 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bdfcd76-3dd3-46fd-a382-0e40e648aa17" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134731 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="e33113d6-3645-47a5-8f09-edee50ffb54d" containerName="maas-api" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134738 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3fc56a-e7a0-445a-b206-f903fd3d12d9" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134744 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="1edcbec2-0930-4b49-b14a-2d21568b5318" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134749 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="662bceeb-beae-4a4f-b08a-08706bed2d2e" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134755 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="eda7b470-9598-458c-a270-4b0789924f39" containerName="istio-proxy" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134760 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e4f9f6-f644-40b2-93fb-0acb72d86443" containerName="istio-proxy" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134769 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="679af1f2-dd8a-4af5-a8d8-518a4a0db304" containerName="maas-api" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134773 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa8baac2-125d-42f6-8449-5fd590bf4b2b" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134779 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f5464f-5d94-499e-9fee-7553aa1e78a4" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134784 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b006c9-5373-4b54-b9a3-ae0ea5425005" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134789 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5fac629-6443-4e4e-a5b3-08a949199c32" containerName="manager" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134794 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="0705edee-7b08-4a0d-b47c-7b8ce7f3279a" containerName="maas-api" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134799 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48999ea-6a3b-427c-a22b-621a5cc997db" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134805 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ce93966-0d0b-4b1d-aa96-16315a7467b5" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134810 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="58309a66-dfdf-4793-bbc5-5c7b33ca111e" containerName="payload-processing" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134815 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe1449b-31f2-45a8-b054-d4dc8ec7e72d" containerName="istio-proxy" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134821 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="62344e41-1f57-4ca4-9805-668cec58dd3b" containerName="maas-api" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134827 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="4524d0a6-727b-47a9-b3b9-e83e5cf3a52d" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134832 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae009e3-4ccc-46ef-895c-5ecc6ea4e599" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134839 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9455eb-457d-4fec-aca7-f55202209d2c" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134844 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eef8d7a-0034-40d0-9027-d6c4528dd24e" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134849 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d38fac8-2315-4693-8dae-4e449356ee4c" containerName="main" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134855 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9cd8ca-bd80-40fb-b420-229f3e6fcd2a" containerName="maas-api" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134860 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ade2d0-82c2-4a55-9bfa-d6f5916c81fd" containerName="istio-proxy" Nov 03 22:11:43.135818 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.134868 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="9204dab2-ccbe-4405-b9c7-666f23309bbd" containerName="payload-pre-processing" Nov 03 22:11:43.138021 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.137993 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.205029 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.204985 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr6vz\" (UniqueName: \"kubernetes.io/projected/09e130a7-702e-4a97-b02f-ca68a03fa694-kube-api-access-mr6vz\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.205225 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.205106 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e130a7-702e-4a97-b02f-ca68a03fa694-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.205225 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.205140 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09e130a7-702e-4a97-b02f-ca68a03fa694-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.205225 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.205171 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09e130a7-702e-4a97-b02f-ca68a03fa694-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.205225 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.205193 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e130a7-702e-4a97-b02f-ca68a03fa694-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.205408 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.205333 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09e130a7-702e-4a97-b02f-ca68a03fa694-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.205408 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.205360 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09e130a7-702e-4a97-b02f-ca68a03fa694-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.205484 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.205460 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09e130a7-702e-4a97-b02f-ca68a03fa694-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.205522 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.205494 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09e130a7-702e-4a97-b02f-ca68a03fa694-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.237685 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.237651 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8"] Nov 03 22:11:43.306627 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.306585 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09e130a7-702e-4a97-b02f-ca68a03fa694-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.306627 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.306631 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09e130a7-702e-4a97-b02f-ca68a03fa694-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.306954 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.306668 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09e130a7-702e-4a97-b02f-ca68a03fa694-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.306954 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.306684 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09e130a7-702e-4a97-b02f-ca68a03fa694-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.306954 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.306715 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr6vz\" (UniqueName: \"kubernetes.io/projected/09e130a7-702e-4a97-b02f-ca68a03fa694-kube-api-access-mr6vz\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.306954 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.306744 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e130a7-702e-4a97-b02f-ca68a03fa694-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.306954 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.306761 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09e130a7-702e-4a97-b02f-ca68a03fa694-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.306954 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.306801 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09e130a7-702e-4a97-b02f-ca68a03fa694-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.306954 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.306831 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e130a7-702e-4a97-b02f-ca68a03fa694-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.307347 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.307222 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09e130a7-702e-4a97-b02f-ca68a03fa694-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.307347 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.307296 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e130a7-702e-4a97-b02f-ca68a03fa694-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.307450 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.307429 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e130a7-702e-4a97-b02f-ca68a03fa694-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.307496 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.307456 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09e130a7-702e-4a97-b02f-ca68a03fa694-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.307535 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.307500 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09e130a7-702e-4a97-b02f-ca68a03fa694-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.309273 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.309252 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09e130a7-702e-4a97-b02f-ca68a03fa694-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.309422 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.309401 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09e130a7-702e-4a97-b02f-ca68a03fa694-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.365379 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.365349 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09e130a7-702e-4a97-b02f-ca68a03fa694-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.365704 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.365676 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr6vz\" (UniqueName: \"kubernetes.io/projected/09e130a7-702e-4a97-b02f-ca68a03fa694-kube-api-access-mr6vz\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b8kl8\" (UID: \"09e130a7-702e-4a97-b02f-ca68a03fa694\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.451764 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.451729 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:43.877174 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.877089 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8"] Nov 03 22:11:43.883116 ip-10-0-130-134 kubenswrapper[4963]: W1103 22:11:43.883072 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e130a7_702e_4a97_b02f_ca68a03fa694.slice/crio-95cc00b2f75094aff56673c96837c2d9dd35e0f8014e60b0d7b485739716b384 WatchSource:0}: Error finding container 95cc00b2f75094aff56673c96837c2d9dd35e0f8014e60b0d7b485739716b384: Status 404 returned error can't find the container with id 95cc00b2f75094aff56673c96837c2d9dd35e0f8014e60b0d7b485739716b384 Nov 03 22:11:43.885201 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.885170 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:11:43.885279 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.885236 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:11:43.885279 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:43.885262 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:11:44.337891 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:44.337849 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" event={"ID":"09e130a7-702e-4a97-b02f-ca68a03fa694","Type":"ContainerStarted","Data":"7553f976ebe801f6af61c33f04f13fe3eee30d20245d7305c413870dcd979400"} Nov 03 22:11:44.338207 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:44.338184 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" event={"ID":"09e130a7-702e-4a97-b02f-ca68a03fa694","Type":"ContainerStarted","Data":"95cc00b2f75094aff56673c96837c2d9dd35e0f8014e60b0d7b485739716b384"} Nov 03 22:11:44.452858 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:44.452820 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:44.454177 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:44.454151 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:11:44.454310 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:44.454210 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" podUID="09e130a7-702e-4a97-b02f-ca68a03fa694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:11:44.555026 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:44.554968 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" podStartSLOduration=1.554953227 podStartE2EDuration="1.554953227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:11:44.511246716 +0000 UTC m=+5053.671318988" watchObservedRunningTime="2026-11-03 22:11:44.554953227 +0000 UTC m=+5053.715025499" Nov 03 22:11:45.453118 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:45.453082 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:11:45.453549 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:45.453135 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" podUID="09e130a7-702e-4a97-b02f-ca68a03fa694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:11:46.453064 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:46.453028 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:11:46.453261 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:46.453081 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" podUID="09e130a7-702e-4a97-b02f-ca68a03fa694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:11:47.200007 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:47.196464 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:47.453064 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:47.452966 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:11:47.453064 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:47.453029 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" podUID="09e130a7-702e-4a97-b02f-ca68a03fa694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:11:48.452415 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:48.452377 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:11:48.452825 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:48.452431 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" podUID="09e130a7-702e-4a97-b02f-ca68a03fa694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:11:49.366211 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:49.366176 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:49.452411 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:49.452369 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:11:49.452600 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:49.452440 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" podUID="09e130a7-702e-4a97-b02f-ca68a03fa694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:11:49.551736 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:49.551701 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:50.452427 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:50.452386 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:11:50.452628 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:50.452471 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" podUID="09e130a7-702e-4a97-b02f-ca68a03fa694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:11:51.452740 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:51.452700 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:11:51.453185 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:51.452770 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" podUID="09e130a7-702e-4a97-b02f-ca68a03fa694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:11:52.456486 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:52.456456 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:52.456931 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:52.456714 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:52.457499 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:52.457478 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" Nov 03 22:11:59.422104 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:59.422068 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:11:59.602605 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:11:59.602565 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:12.527745 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:12:12.527664 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:12:28.415358 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.415320 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m"] Nov 03 22:12:28.419195 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.419173 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.548449 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.548406 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb6adca-8296-435f-96bc-90250221f784-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.548449 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.548451 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb6adca-8296-435f-96bc-90250221f784-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.548687 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.548542 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb6adca-8296-435f-96bc-90250221f784-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.548687 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.548579 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb6adca-8296-435f-96bc-90250221f784-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.548687 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.548645 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb6adca-8296-435f-96bc-90250221f784-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.548790 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.548700 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb6adca-8296-435f-96bc-90250221f784-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.548790 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.548724 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb6adca-8296-435f-96bc-90250221f784-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.548790 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.548763 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb6adca-8296-435f-96bc-90250221f784-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.548941 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.548886 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv2pv\" (UniqueName: \"kubernetes.io/projected/bbb6adca-8296-435f-96bc-90250221f784-kube-api-access-wv2pv\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.618049 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.618009 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m"] Nov 03 22:12:28.649767 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.649645 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2pv\" (UniqueName: \"kubernetes.io/projected/bbb6adca-8296-435f-96bc-90250221f784-kube-api-access-wv2pv\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.649767 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.649694 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb6adca-8296-435f-96bc-90250221f784-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.649767 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.649724 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb6adca-8296-435f-96bc-90250221f784-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.649767 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.649766 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb6adca-8296-435f-96bc-90250221f784-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.656286 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.649790 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb6adca-8296-435f-96bc-90250221f784-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.656286 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.649824 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb6adca-8296-435f-96bc-90250221f784-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.656286 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.649862 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb6adca-8296-435f-96bc-90250221f784-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.656286 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.649891 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb6adca-8296-435f-96bc-90250221f784-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.656286 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.649969 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb6adca-8296-435f-96bc-90250221f784-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.656286 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.650296 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb6adca-8296-435f-96bc-90250221f784-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.656286 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.650364 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb6adca-8296-435f-96bc-90250221f784-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.656286 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.650378 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb6adca-8296-435f-96bc-90250221f784-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.656286 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.650427 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb6adca-8296-435f-96bc-90250221f784-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.656286 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.650595 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb6adca-8296-435f-96bc-90250221f784-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.656286 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.652358 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb6adca-8296-435f-96bc-90250221f784-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.656286 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.652598 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb6adca-8296-435f-96bc-90250221f784-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.708625 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.708587 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb6adca-8296-435f-96bc-90250221f784-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.708844 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.708818 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv2pv\" (UniqueName: \"kubernetes.io/projected/bbb6adca-8296-435f-96bc-90250221f784-kube-api-access-wv2pv\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-xcb6m\" (UID: \"bbb6adca-8296-435f-96bc-90250221f784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:28.733143 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:28.733101 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:29.162555 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:29.162468 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m"] Nov 03 22:12:29.206074 ip-10-0-130-134 kubenswrapper[4963]: W1103 22:12:29.206045 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb6adca_8296_435f_96bc_90250221f784.slice/crio-935f8526bba3efef8ff3bf711e125377d82e6888ff26a240ec26d9c166ed1886 WatchSource:0}: Error finding container 935f8526bba3efef8ff3bf711e125377d82e6888ff26a240ec26d9c166ed1886: Status 404 returned error can't find the container with id 935f8526bba3efef8ff3bf711e125377d82e6888ff26a240ec26d9c166ed1886 Nov 03 22:12:29.208354 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:29.208318 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:12:29.208445 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:29.208390 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:12:29.208445 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:29.208424 4963 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:12:29.567206 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:29.567166 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" event={"ID":"bbb6adca-8296-435f-96bc-90250221f784","Type":"ContainerStarted","Data":"5a7e5a8275594f8dbe20f31141feadc19feac35ad2a296b3bfea8ad1522935e0"} Nov 03 22:12:29.567644 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:29.567232 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" event={"ID":"bbb6adca-8296-435f-96bc-90250221f784","Type":"ContainerStarted","Data":"935f8526bba3efef8ff3bf711e125377d82e6888ff26a240ec26d9c166ed1886"} Nov 03 22:12:29.734304 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:29.734254 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:30.735287 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:30.735250 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:12:30.735791 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:30.735309 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" podUID="bbb6adca-8296-435f-96bc-90250221f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:12:31.667684 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:31.667657 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 22:12:31.715640 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:31.715612 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 22:12:31.733565 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:31.733530 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:12:31.733734 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:31.733580 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" podUID="bbb6adca-8296-435f-96bc-90250221f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:12:32.734061 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:32.734019 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:12:32.734486 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:32.734077 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" podUID="bbb6adca-8296-435f-96bc-90250221f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:12:33.734251 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:33.734212 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:12:33.734672 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:33.734272 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" podUID="bbb6adca-8296-435f-96bc-90250221f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:12:34.669150 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:34.669111 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:34.734329 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:34.734287 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:12:34.734724 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:34.734341 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" podUID="bbb6adca-8296-435f-96bc-90250221f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:12:35.734362 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:35.734316 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:12:35.734777 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:35.734386 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" podUID="bbb6adca-8296-435f-96bc-90250221f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:12:36.385058 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:36.385022 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:36.733936 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:36.733865 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:12:36.734176 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:36.733973 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" podUID="bbb6adca-8296-435f-96bc-90250221f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:12:38.775091 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:38.775050 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:12:38.775599 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:38.775117 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" podUID="bbb6adca-8296-435f-96bc-90250221f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:12:39.809506 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:39.809465 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:12:39.809960 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:39.809532 4963 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" podUID="bbb6adca-8296-435f-96bc-90250221f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:12:39.823319 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:39.823276 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:39.823688 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:39.823667 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:39.824540 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:39.824519 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" Nov 03 22:12:39.979979 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:39.979927 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" podStartSLOduration=11.97989205 podStartE2EDuration="11.97989205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:29.694762257 +0000 UTC m=+5098.854834606" watchObservedRunningTime="2026-11-03 22:12:39.97989205 +0000 UTC m=+5109.139964322" Nov 03 22:12:42.569522 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:12:42.569484 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:12:49.906954 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:49.901892 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:57.368127 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:57.368085 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:59.133540 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:12:59.133507 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:12.529051 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:13:12.529006 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:13:42.585400 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:13:42.585156 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:13:43.590622 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:13:43.590591 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:44.264942 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:13:44.264885 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:50.320728 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:13:50.320631 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:51.962346 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:13:51.962305 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:57.308801 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:13:57.308762 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:13:57.489980 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:13:57.489946 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:12.527667 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:14:12.527627 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:14:24.878777 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:14:24.878721 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:42.528101 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:14:42.528060 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:14:50.690058 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:14:50.690021 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:14:50.877045 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:14:50.877008 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:08.942048 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:08.942011 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:09.602063 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:09.602026 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:09.792155 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:09.792123 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:12.528915 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:15:12.528819 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:15:14.421929 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:14.421582 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:14.650776 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:14.650437 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:14.849928 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:14.842691 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:16.637941 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:16.637888 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:17.796280 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:17.796237 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:17.983048 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:17.983013 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:20.832536 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:20.832495 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh"] Nov 03 22:15:20.836328 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:20.836305 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:15:20.933765 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:20.933725 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-60736a-serving-cert\"" Nov 03 22:15:20.933995 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:20.933766 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 22:15:20.933995 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:20.933774 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:15:20.933995 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:20.933854 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6whcf\"" Nov 03 22:15:20.933995 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:20.933890 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:15:20.935926 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:20.935891 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh"] Nov 03 22:15:21.008858 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:21.008824 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4s9k\" (UniqueName: \"kubernetes.io/projected/1f30a024-5546-45cd-a6c9-7badf95c59b3-kube-api-access-p4s9k\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh\" (UID: \"1f30a024-5546-45cd-a6c9-7badf95c59b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:15:21.009080 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:21.008887 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f30a024-5546-45cd-a6c9-7badf95c59b3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh\" (UID: \"1f30a024-5546-45cd-a6c9-7badf95c59b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:15:21.009080 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:21.009053 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f30a024-5546-45cd-a6c9-7badf95c59b3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh\" (UID: \"1f30a024-5546-45cd-a6c9-7badf95c59b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:15:21.009204 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:21.009096 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f30a024-5546-45cd-a6c9-7badf95c59b3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh\" (UID: \"1f30a024-5546-45cd-a6c9-7badf95c59b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:15:21.071161 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:21.071127 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:21.110476 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:21.110398 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f30a024-5546-45cd-a6c9-7badf95c59b3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh\" (UID: \"1f30a024-5546-45cd-a6c9-7badf95c59b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:15:21.110644 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:21.110488 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f30a024-5546-45cd-a6c9-7badf95c59b3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh\" (UID: \"1f30a024-5546-45cd-a6c9-7badf95c59b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:15:21.110644 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:21.110519 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f30a024-5546-45cd-a6c9-7badf95c59b3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh\" (UID: \"1f30a024-5546-45cd-a6c9-7badf95c59b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:15:21.110644 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:21.110537 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4s9k\" (UniqueName: \"kubernetes.io/projected/1f30a024-5546-45cd-a6c9-7badf95c59b3-kube-api-access-p4s9k\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh\" (UID: \"1f30a024-5546-45cd-a6c9-7badf95c59b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:15:21.110644 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:15:21.110543 4963 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-serving-cert: secret "maas-api-e2e-worker-single-main-60736a-serving-cert" not found Nov 03 22:15:21.110644 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:15:21.110618 4963 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f30a024-5546-45cd-a6c9-7badf95c59b3-maas-api-tls podName:1f30a024-5546-45cd-a6c9-7badf95c59b3 nodeName:}" failed. No retries permitted until 2026-11-03 22:15:21.610602323 +0000 UTC m=+5270.770674573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f30a024-5546-45cd-a6c9-7badf95c59b3-maas-api-tls") pod "maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" (UID: "1f30a024-5546-45cd-a6c9-7badf95c59b3") : secret "maas-api-e2e-worker-single-main-60736a-serving-cert" not found Nov 03 22:15:21.111232 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:21.111204 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f30a024-5546-45cd-a6c9-7badf95c59b3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh\" (UID: \"1f30a024-5546-45cd-a6c9-7badf95c59b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:15:21.113139 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:21.113118 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f30a024-5546-45cd-a6c9-7badf95c59b3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh\" (UID: \"1f30a024-5546-45cd-a6c9-7badf95c59b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:15:21.171726 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:21.171682 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4s9k\" (UniqueName: \"kubernetes.io/projected/1f30a024-5546-45cd-a6c9-7badf95c59b3-kube-api-access-p4s9k\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh\" (UID: \"1f30a024-5546-45cd-a6c9-7badf95c59b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:15:21.253041 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:21.253002 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:21.441014 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:21.440930 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:21.614436 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:21.614402 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f30a024-5546-45cd-a6c9-7badf95c59b3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh\" (UID: \"1f30a024-5546-45cd-a6c9-7badf95c59b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:15:21.616930 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:21.616874 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f30a024-5546-45cd-a6c9-7badf95c59b3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh\" (UID: \"1f30a024-5546-45cd-a6c9-7badf95c59b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:15:21.749750 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:21.749707 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:15:22.122541 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:22.122448 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh"] Nov 03 22:15:22.128364 ip-10-0-130-134 kubenswrapper[4963]: W1103 22:15:22.128333 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f30a024_5546_45cd_a6c9_7badf95c59b3.slice/crio-6e732bf3bc9305e168f75fda5a0d81cbf2adcf779cb019417a8f825dc94ba367 WatchSource:0}: Error finding container 6e732bf3bc9305e168f75fda5a0d81cbf2adcf779cb019417a8f825dc94ba367: Status 404 returned error can't find the container with id 6e732bf3bc9305e168f75fda5a0d81cbf2adcf779cb019417a8f825dc94ba367 Nov 03 22:15:22.129878 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:22.129854 4963 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:15:22.606705 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:22.606659 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" event={"ID":"1f30a024-5546-45cd-a6c9-7badf95c59b3","Type":"ContainerStarted","Data":"6e732bf3bc9305e168f75fda5a0d81cbf2adcf779cb019417a8f825dc94ba367"} Nov 03 22:15:23.613225 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:23.613182 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" event={"ID":"1f30a024-5546-45cd-a6c9-7badf95c59b3","Type":"ContainerStarted","Data":"f2cf4414ca0f07e957685e4ae15a56bbe24e89ba73e30b9e6b668db8cce9449c"} Nov 03 22:15:23.613720 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:23.613319 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:15:23.727096 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:23.727045 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" podStartSLOduration=2.781899867 podStartE2EDuration="3.727031687s" totalImagesPullingTime="945.13182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:23.725057009 +0000 UTC m=+5272.885129284" watchObservedRunningTime="2026-11-03 22:15:23.727031687 +0000 UTC m=+5272.887103958" Nov 03 22:15:26.221919 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:26.221871 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:29.624000 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:29.623970 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:15:31.340658 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:31.340619 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:36.238446 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:36.238412 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:39.771262 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:39.771229 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:40.003059 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:40.003028 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:41.763633 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:41.763592 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:42.529492 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:15:42.529443 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:15:42.648773 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:42.648739 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:46.829757 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:46.829724 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:51.002127 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:51.002083 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:51.837459 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:51.837415 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:15:56.833660 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:15:56.833627 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:01.838459 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:01.838426 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:06.859383 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:06.859349 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:11.735577 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:11.735546 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:12.528172 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:16:12.528130 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:16:16.676700 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:16.676667 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:21.726179 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:21.726135 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:26.732626 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:26.732591 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:31.734721 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:31.734677 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:33.779821 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:33.779776 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:33.967593 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:33.967563 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:34.154716 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:34.154630 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:34.436850 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:34.436775 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:36.739916 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:36.739881 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:39.018997 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:39.018967 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:40.596660 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:40.596590 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:41.031793 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:41.031755 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh"] Nov 03 22:16:41.032019 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:41.031978 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" podUID="1f30a024-5546-45cd-a6c9-7badf95c59b3" containerName="maas-api" containerID="cri-o://f2cf4414ca0f07e957685e4ae15a56bbe24e89ba73e30b9e6b668db8cce9449c" gracePeriod=30 Nov 03 22:16:41.176441 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:41.176408 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:41.537034 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:41.536995 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:41.712526 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:41.712494 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:42.289093 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:42.289066 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:16:42.377878 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:42.377781 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f30a024-5546-45cd-a6c9-7badf95c59b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f30a024-5546-45cd-a6c9-7badf95c59b3-maas-api-tls\") pod \"1f30a024-5546-45cd-a6c9-7badf95c59b3\" (UID: \"1f30a024-5546-45cd-a6c9-7badf95c59b3\") " Nov 03 22:16:42.377878 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:42.377826 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f30a024-5546-45cd-a6c9-7badf95c59b3-kube-api-access-p4s9k\" (UniqueName: \"kubernetes.io/projected/1f30a024-5546-45cd-a6c9-7badf95c59b3-kube-api-access-p4s9k\") pod \"1f30a024-5546-45cd-a6c9-7badf95c59b3\" (UID: \"1f30a024-5546-45cd-a6c9-7badf95c59b3\") " Nov 03 22:16:42.378139 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:42.377964 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f30a024-5546-45cd-a6c9-7badf95c59b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f30a024-5546-45cd-a6c9-7badf95c59b3-metrics-certs\") pod \"1f30a024-5546-45cd-a6c9-7badf95c59b3\" (UID: \"1f30a024-5546-45cd-a6c9-7badf95c59b3\") " Nov 03 22:16:42.378139 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:42.377988 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f30a024-5546-45cd-a6c9-7badf95c59b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f30a024-5546-45cd-a6c9-7badf95c59b3-openshift-service-ca\") pod \"1f30a024-5546-45cd-a6c9-7badf95c59b3\" (UID: \"1f30a024-5546-45cd-a6c9-7badf95c59b3\") " Nov 03 22:16:42.378464 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:42.378436 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f30a024-5546-45cd-a6c9-7badf95c59b3-openshift-service-ca" pod "1f30a024-5546-45cd-a6c9-7badf95c59b3" (UID: "1f30a024-5546-45cd-a6c9-7badf95c59b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:16:42.380048 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:42.380023 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f30a024-5546-45cd-a6c9-7badf95c59b3-maas-api-tls" pod "1f30a024-5546-45cd-a6c9-7badf95c59b3" (UID: "1f30a024-5546-45cd-a6c9-7badf95c59b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:16:42.380136 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:42.380057 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f30a024-5546-45cd-a6c9-7badf95c59b3-metrics-certs" pod "1f30a024-5546-45cd-a6c9-7badf95c59b3" (UID: "1f30a024-5546-45cd-a6c9-7badf95c59b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:16:42.380186 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:42.380165 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f30a024-5546-45cd-a6c9-7badf95c59b3-kube-api-access-p4s9k" pod "1f30a024-5546-45cd-a6c9-7badf95c59b3" (UID: "1f30a024-5546-45cd-a6c9-7badf95c59b3"). InnerVolumeSpecName "kube-api-access-p4s9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:16:42.478817 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:42.478776 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f30a024-5546-45cd-a6c9-7badf95c59b3-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:16:42.478817 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:42.478811 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f30a024-5546-45cd-a6c9-7badf95c59b3-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:16:42.478817 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:42.478820 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4s9k\" (UniqueName: \"kubernetes.io/projected/1f30a024-5546-45cd-a6c9-7badf95c59b3-kube-api-access-p4s9k\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:16:42.479104 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:42.478829 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f30a024-5546-45cd-a6c9-7badf95c59b3-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:16:42.528684 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:16:42.528648 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:16:43.014956 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:43.014921 4963 generic.go:356] "Generic (PLEG): container finished" podID="1f30a024-5546-45cd-a6c9-7badf95c59b3" containerID="f2cf4414ca0f07e957685e4ae15a56bbe24e89ba73e30b9e6b668db8cce9449c" exitCode=0 Nov 03 22:16:43.015434 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:43.014970 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" Nov 03 22:16:43.015434 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:43.015007 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" event={"ID":"1f30a024-5546-45cd-a6c9-7badf95c59b3","Type":"ContainerDied","Data":"f2cf4414ca0f07e957685e4ae15a56bbe24e89ba73e30b9e6b668db8cce9449c"} Nov 03 22:16:43.015434 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:43.015068 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh" event={"ID":"1f30a024-5546-45cd-a6c9-7badf95c59b3","Type":"ContainerDied","Data":"6e732bf3bc9305e168f75fda5a0d81cbf2adcf779cb019417a8f825dc94ba367"} Nov 03 22:16:43.015434 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:43.015095 4963 scope.go:122] "RemoveContainer" containerID="f2cf4414ca0f07e957685e4ae15a56bbe24e89ba73e30b9e6b668db8cce9449c" Nov 03 22:16:43.024373 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:43.024349 4963 scope.go:122] "RemoveContainer" containerID="f2cf4414ca0f07e957685e4ae15a56bbe24e89ba73e30b9e6b668db8cce9449c" Nov 03 22:16:43.024684 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:16:43.024660 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2cf4414ca0f07e957685e4ae15a56bbe24e89ba73e30b9e6b668db8cce9449c\": container with ID starting with f2cf4414ca0f07e957685e4ae15a56bbe24e89ba73e30b9e6b668db8cce9449c not found: ID does not exist" containerID="f2cf4414ca0f07e957685e4ae15a56bbe24e89ba73e30b9e6b668db8cce9449c" Nov 03 22:16:43.024734 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:43.024693 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2cf4414ca0f07e957685e4ae15a56bbe24e89ba73e30b9e6b668db8cce9449c"} err="failed to get container status \"f2cf4414ca0f07e957685e4ae15a56bbe24e89ba73e30b9e6b668db8cce9449c\": rpc error: code = NotFound desc = could not find container \"f2cf4414ca0f07e957685e4ae15a56bbe24e89ba73e30b9e6b668db8cce9449c\": container with ID starting with f2cf4414ca0f07e957685e4ae15a56bbe24e89ba73e30b9e6b668db8cce9449c not found: ID does not exist" Nov 03 22:16:43.171982 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:43.171946 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh"] Nov 03 22:16:43.219510 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:43.219477 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-60736a-6fc6fc4b6f-8bqmh"] Nov 03 22:16:44.965966 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:44.965929 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f30a024-5546-45cd-a6c9-7badf95c59b3" path="/var/lib/kubelet/pods/1f30a024-5546-45cd-a6c9-7badf95c59b3/volumes" Nov 03 22:16:46.460251 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:46.460220 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:46.639825 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:46.639791 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:16:51.338040 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:16:51.338007 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:12.527736 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:17:12.527697 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:17:15.718239 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:15.718201 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:15.920244 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:15.920210 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:16.554710 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:16.554674 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:17.826436 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:17.826402 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:18.185489 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:18.185403 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:18.368305 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:18.368275 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:18.585849 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:18.585819 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:22.385660 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:22.385624 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:22.569692 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:22.569660 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:23.331082 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:23.331050 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:28.332639 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:28.332604 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:31.699293 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:31.699264 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 22:17:31.752041 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:31.752013 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 22:17:33.399657 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:33.399624 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:38.334077 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:38.334045 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:42.527745 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:17:42.527705 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:17:43.183009 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:43.182975 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:47.660627 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:47.660589 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:49.277209 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:49.277174 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:49.565114 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:49.565033 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:50.188131 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:50.188097 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:51.690431 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:51.690402 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:17:57.196811 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:17:57.196776 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:12.526600 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:18:12.526519 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:18:24.782963 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:18:24.782926 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:18:42.528573 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:18:42.528537 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:18:52.081161 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:18:52.081126 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:11.505701 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:19:11.505666 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:12.527358 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:19:12.527320 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:19:26.302357 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:19:26.302313 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:19:42.528604 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:19:42.528514 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:19:52.376967 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:19:52.376895 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:12.527712 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:20:12.527669 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:20:38.324318 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:20:38.324278 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:42.527435 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:20:42.527399 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:20:52.704884 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:20:52.704847 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:57.093091 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:20:57.093043 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:20:57.536342 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:20:57.536311 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:00.277478 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:21:00.277438 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:12.527747 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:21:12.527663 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:21:28.232124 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:21:28.232090 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:28.416761 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:21:28.416723 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:42.527996 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:21:42.527956 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:21:53.045552 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:21:53.045517 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:09.797740 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:22:09.797698 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:12.527319 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:22:12.527282 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:22:31.729990 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:22:31.729876 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 22:22:31.794165 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:22:31.794138 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 22:22:37.911371 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:22:37.911335 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:38.096274 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:22:38.096240 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:42.528332 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:22:42.528243 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:22:53.382830 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:22:53.382784 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:57.178253 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:22:57.178216 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:57.660781 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:22:57.660746 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:58.835367 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:22:58.835332 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:12.573960 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:23:12.573895 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:23:40.159794 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:23:40.159761 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:42.527179 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:23:42.527142 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:23:50.797096 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:23:50.797061 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:53.765426 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:23:53.765392 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:12.532438 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:24:12.532341 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:24:42.527566 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:24:42.527527 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:24:54.063270 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:24:54.063200 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:12.541378 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:25:12.541338 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:25:42.527320 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:25:42.527231 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:25:54.399077 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:25:54.399035 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:12.528668 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:26:12.528630 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:26:42.532531 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:26:42.532483 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:26:54.742848 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:26:54.742812 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:12.528343 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:27:12.528260 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:27:31.762811 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:27:31.762703 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 22:27:31.826450 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:27:31.826423 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 22:27:42.527996 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:27:42.527957 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:27:55.086892 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:27:55.086837 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:12.527549 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:28:12.527503 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:28:17.272553 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:17.272518 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:19.343508 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:19.343475 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:23.575982 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:23.575947 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:23.774528 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:23.774496 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:23.984220 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:23.984184 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:25.727019 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:25.726987 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:26.893819 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:26.893778 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:27.081913 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:27.081863 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:30.365839 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:30.365800 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:30.551443 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:30.551412 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:30.735533 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:30.735502 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:35.453932 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:35.453827 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:40.575003 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:40.574899 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:42.527467 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:28:42.527419 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:28:45.521568 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:45.521525 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:48.919634 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:48.919603 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:49.180579 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:49.180474 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:50.968500 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:50.968453 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:51.984062 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:51.984020 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j"] Nov 03 22:28:51.984671 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:51.984441 4963 cpu_manager.go:414] "containerMap: removing container" podUID="1f30a024-5546-45cd-a6c9-7badf95c59b3" containerName="maas-api" Nov 03 22:28:51.984671 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:51.984452 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f30a024-5546-45cd-a6c9-7badf95c59b3" containerName="maas-api" Nov 03 22:28:51.984671 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:51.984528 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f30a024-5546-45cd-a6c9-7badf95c59b3" containerName="maas-api" Nov 03 22:28:51.987741 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:51.987721 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.044023 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.043984 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j"] Nov 03 22:28:52.088337 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.088304 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-home\") pod \"simulator-single-main-51b683-kserve-bcd65667c-dvj6j\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.088533 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.088354 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-tmp-dir\") pod \"simulator-single-main-51b683-kserve-bcd65667c-dvj6j\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.088533 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.088433 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-dshm\") pod \"simulator-single-main-51b683-kserve-bcd65667c-dvj6j\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.088671 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.088542 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pncdx\" (UniqueName: \"kubernetes.io/projected/c1503b13-6950-4c48-bb71-e0794d354a45-kube-api-access-pncdx\") pod \"simulator-single-main-51b683-kserve-bcd65667c-dvj6j\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.088671 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.088576 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1503b13-6950-4c48-bb71-e0794d354a45-tls-certs\") pod \"simulator-single-main-51b683-kserve-bcd65667c-dvj6j\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.088811 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.088664 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-model-cache\") pod \"simulator-single-main-51b683-kserve-bcd65667c-dvj6j\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.088811 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.088689 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-51b683\"/\"openshift-service-ca.crt\"" Nov 03 22:28:52.088932 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.088894 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-51b683\"/\"kube-root-ca.crt\"" Nov 03 22:28:52.091010 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.090988 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-51b683\"/\"default-dockercfg-kbkn6\"" Nov 03 22:28:52.091113 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.091026 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-51b683\"/\"simulator-single-main-51b683-kserve-self-signed-certs\"" Nov 03 22:28:52.141173 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.141133 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:52.189556 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.189518 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-dshm\") pod \"simulator-single-main-51b683-kserve-bcd65667c-dvj6j\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.189749 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.189583 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pncdx\" (UniqueName: \"kubernetes.io/projected/c1503b13-6950-4c48-bb71-e0794d354a45-kube-api-access-pncdx\") pod \"simulator-single-main-51b683-kserve-bcd65667c-dvj6j\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.189749 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.189610 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1503b13-6950-4c48-bb71-e0794d354a45-tls-certs\") pod \"simulator-single-main-51b683-kserve-bcd65667c-dvj6j\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.189749 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.189678 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-model-cache\") pod \"simulator-single-main-51b683-kserve-bcd65667c-dvj6j\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.189967 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.189750 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-home\") pod \"simulator-single-main-51b683-kserve-bcd65667c-dvj6j\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.189967 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.189794 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-tmp-dir\") pod \"simulator-single-main-51b683-kserve-bcd65667c-dvj6j\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.190206 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.190183 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-tmp-dir\") pod \"simulator-single-main-51b683-kserve-bcd65667c-dvj6j\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.190291 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.190228 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-home\") pod \"simulator-single-main-51b683-kserve-bcd65667c-dvj6j\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.190291 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.190245 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-model-cache\") pod \"simulator-single-main-51b683-kserve-bcd65667c-dvj6j\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.191887 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.191867 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-dshm\") pod \"simulator-single-main-51b683-kserve-bcd65667c-dvj6j\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.192306 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.192291 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1503b13-6950-4c48-bb71-e0794d354a45-tls-certs\") pod \"simulator-single-main-51b683-kserve-bcd65667c-dvj6j\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.249559 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.249472 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pncdx\" (UniqueName: \"kubernetes.io/projected/c1503b13-6950-4c48-bb71-e0794d354a45-kube-api-access-pncdx\") pod \"simulator-single-main-51b683-kserve-bcd65667c-dvj6j\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.299775 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.299722 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:52.675098 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:52.675017 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j"] Nov 03 22:28:52.680705 ip-10-0-130-134 kubenswrapper[4963]: W1103 22:28:52.680676 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1503b13_6950_4c48_bb71_e0794d354a45.slice/crio-9e2124e51c28f2ea07d7eeb2345b17d7f13d9311fbfce1fd5e0d43b48026eaea WatchSource:0}: Error finding container 9e2124e51c28f2ea07d7eeb2345b17d7f13d9311fbfce1fd5e0d43b48026eaea: Status 404 returned error can't find the container with id 9e2124e51c28f2ea07d7eeb2345b17d7f13d9311fbfce1fd5e0d43b48026eaea Nov 03 22:28:53.249594 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:53.249558 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" event={"ID":"c1503b13-6950-4c48-bb71-e0794d354a45","Type":"ContainerStarted","Data":"d1eff6c6fdfe79a7e20ce92494ebc65c58ec503658a98fd5d79b1e6ff998ee37"} Nov 03 22:28:53.249594 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:53.249598 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" event={"ID":"c1503b13-6950-4c48-bb71-e0794d354a45","Type":"ContainerStarted","Data":"9e2124e51c28f2ea07d7eeb2345b17d7f13d9311fbfce1fd5e0d43b48026eaea"} Nov 03 22:28:53.299869 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:53.299823 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:28:53.363472 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:53.363418 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" podStartSLOduration=2.363402817 podStartE2EDuration="2.363402817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:53.362243789 +0000 UTC m=+6082.522316061" watchObservedRunningTime="2026-11-03 22:28:53.363402817 +0000 UTC m=+6082.523475088" Nov 03 22:28:55.472968 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:55.472932 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:56.030606 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:28:56.030573 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:01.052170 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:01.052134 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:02.300226 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:02.300184 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:29:02.313159 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:02.313125 4963 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:29:03.312617 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:03.312587 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:29:06.058099 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:06.058060 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:11.051106 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:11.051060 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:12.528013 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:29:12.527967 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:29:16.034253 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:16.034220 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:21.052676 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:21.052642 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:26.133789 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:26.133756 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:31.242007 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:31.241965 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:36.162690 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:36.162649 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:36.344689 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:36.344648 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:41.278240 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:41.278197 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:42.527604 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:29:42.527558 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:29:43.711930 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:43.711886 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:46.234120 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:46.234089 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:48.167366 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:48.167331 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:48.629031 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:48.628994 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:51.221128 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:51.221091 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:53.098418 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:53.098381 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j"] Nov 03 22:29:53.219738 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:53.219708 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:54.632354 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:54.632321 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:55.586315 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:55.586235 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:55.769064 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:55.769032 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:56.336528 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:29:56.336489 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:00.159713 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:00.159680 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g"] Nov 03 22:30:00.163604 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:00.163581 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" Nov 03 22:30:00.213859 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:00.213822 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\"" Nov 03 22:30:00.254886 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:00.254840 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64r7v\" (UniqueName: \"kubernetes.io/projected/230144ed-b05d-467a-8ee8-e630bf660cfd-kube-api-access-64r7v\") pod \"maas-api-key-cleanup-29895750-bsb2g\" (UID: \"230144ed-b05d-467a-8ee8-e630bf660cfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" Nov 03 22:30:00.255080 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:00.255056 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230144ed-b05d-467a-8ee8-e630bf660cfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895750-bsb2g\" (UID: \"230144ed-b05d-467a-8ee8-e630bf660cfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" Nov 03 22:30:00.261141 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:00.261114 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:30:00.261377 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:00.261352 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:30:00.263493 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:00.263469 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g"] Nov 03 22:30:00.356559 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:00.356516 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230144ed-b05d-467a-8ee8-e630bf660cfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895750-bsb2g\" (UID: \"230144ed-b05d-467a-8ee8-e630bf660cfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" Nov 03 22:30:00.356773 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:00.356600 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64r7v\" (UniqueName: \"kubernetes.io/projected/230144ed-b05d-467a-8ee8-e630bf660cfd-kube-api-access-64r7v\") pod \"maas-api-key-cleanup-29895750-bsb2g\" (UID: \"230144ed-b05d-467a-8ee8-e630bf660cfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" Nov 03 22:30:00.357293 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:00.357266 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230144ed-b05d-467a-8ee8-e630bf660cfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895750-bsb2g\" (UID: \"230144ed-b05d-467a-8ee8-e630bf660cfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" Nov 03 22:30:00.418558 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:00.418484 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64r7v\" (UniqueName: \"kubernetes.io/projected/230144ed-b05d-467a-8ee8-e630bf660cfd-kube-api-access-64r7v\") pod \"maas-api-key-cleanup-29895750-bsb2g\" (UID: \"230144ed-b05d-467a-8ee8-e630bf660cfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" Nov 03 22:30:00.477092 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:00.477036 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" Nov 03 22:30:00.510179 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:00.510149 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:00.849186 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:00.849146 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g"] Nov 03 22:30:00.855884 ip-10-0-130-134 kubenswrapper[4963]: W1103 22:30:00.855844 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230144ed_b05d_467a_8ee8_e630bf660cfd.slice/crio-a2494eb9f8dbfec8f5fbe89cf4f5d601057f4f7ab34a37489d8d52396545f4f2 WatchSource:0}: Error finding container a2494eb9f8dbfec8f5fbe89cf4f5d601057f4f7ab34a37489d8d52396545f4f2: Status 404 returned error can't find the container with id a2494eb9f8dbfec8f5fbe89cf4f5d601057f4f7ab34a37489d8d52396545f4f2 Nov 03 22:30:00.883665 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:00.883633 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:01.613200 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:01.613164 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" event={"ID":"230144ed-b05d-467a-8ee8-e630bf660cfd","Type":"ContainerStarted","Data":"fc456019e155baa96aa987f37da2b9f2da4cb4126bcde80dfc89b5e375d6031d"} Nov 03 22:30:01.613629 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:01.613223 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" event={"ID":"230144ed-b05d-467a-8ee8-e630bf660cfd","Type":"ContainerStarted","Data":"a2494eb9f8dbfec8f5fbe89cf4f5d601057f4f7ab34a37489d8d52396545f4f2"} Nov 03 22:30:01.730651 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:01.730596 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" podStartSLOduration=1.730579055 podStartE2EDuration="1.730579055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:01.728530474 +0000 UTC m=+6150.888602749" watchObservedRunningTime="2026-11-03 22:30:01.730579055 +0000 UTC m=+6150.890651327" Nov 03 22:30:02.620196 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:02.620150 4963 generic.go:356] "Generic (PLEG): container finished" podID="230144ed-b05d-467a-8ee8-e630bf660cfd" containerID="fc456019e155baa96aa987f37da2b9f2da4cb4126bcde80dfc89b5e375d6031d" exitCode=60 Nov 03 22:30:02.620628 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:02.620236 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" event={"ID":"230144ed-b05d-467a-8ee8-e630bf660cfd","Type":"ContainerDied","Data":"fc456019e155baa96aa987f37da2b9f2da4cb4126bcde80dfc89b5e375d6031d"} Nov 03 22:30:02.620628 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:02.620490 4963 scope.go:122] "RemoveContainer" containerID="fc456019e155baa96aa987f37da2b9f2da4cb4126bcde80dfc89b5e375d6031d" Nov 03 22:30:03.627626 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:03.627588 4963 generic.go:356] "Generic (PLEG): container finished" podID="230144ed-b05d-467a-8ee8-e630bf660cfd" containerID="4281c7da6f89292c99b9ebefb6c359ab65b185d15e8b4a2c4b6e2e42f34b7e33" exitCode=60 Nov 03 22:30:03.628137 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:03.627666 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" event={"ID":"230144ed-b05d-467a-8ee8-e630bf660cfd","Type":"ContainerDied","Data":"4281c7da6f89292c99b9ebefb6c359ab65b185d15e8b4a2c4b6e2e42f34b7e33"} Nov 03 22:30:03.628137 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:03.627742 4963 scope.go:122] "RemoveContainer" containerID="fc456019e155baa96aa987f37da2b9f2da4cb4126bcde80dfc89b5e375d6031d" Nov 03 22:30:03.628137 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:03.628020 4963 scope.go:122] "RemoveContainer" containerID="4281c7da6f89292c99b9ebefb6c359ab65b185d15e8b4a2c4b6e2e42f34b7e33" Nov 03 22:30:03.628341 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:30:03.628316 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895750-bsb2g_odh-ai-gateway-infra(230144ed-b05d-467a-8ee8-e630bf660cfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" podUID="230144ed-b05d-467a-8ee8-e630bf660cfd" Nov 03 22:30:04.635150 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:04.635123 4963 scope.go:122] "RemoveContainer" containerID="4281c7da6f89292c99b9ebefb6c359ab65b185d15e8b4a2c4b6e2e42f34b7e33" Nov 03 22:30:04.635569 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:30:04.635355 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895750-bsb2g_odh-ai-gateway-infra(230144ed-b05d-467a-8ee8-e630bf660cfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" podUID="230144ed-b05d-467a-8ee8-e630bf660cfd" Nov 03 22:30:06.093685 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:06.093638 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:06.283916 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:06.283857 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:08.118726 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.118661 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" podUID="c1503b13-6950-4c48-bb71-e0794d354a45" containerName="main" containerID="cri-o://d1eff6c6fdfe79a7e20ce92494ebc65c58ec503658a98fd5d79b1e6ff998ee37" gracePeriod=45 Nov 03 22:30:08.374590 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.374525 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:30:08.540999 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.540964 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-tmp-dir\") pod \"c1503b13-6950-4c48-bb71-e0794d354a45\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " Nov 03 22:30:08.541186 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.541080 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-model-cache\") pod \"c1503b13-6950-4c48-bb71-e0794d354a45\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " Nov 03 22:30:08.541186 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.541114 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-dshm\") pod \"c1503b13-6950-4c48-bb71-e0794d354a45\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " Nov 03 22:30:08.541186 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.541147 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-home\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-home\") pod \"c1503b13-6950-4c48-bb71-e0794d354a45\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " Nov 03 22:30:08.541186 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.541186 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1503b13-6950-4c48-bb71-e0794d354a45-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1503b13-6950-4c48-bb71-e0794d354a45-tls-certs\") pod \"c1503b13-6950-4c48-bb71-e0794d354a45\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " Nov 03 22:30:08.541436 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.541206 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1503b13-6950-4c48-bb71-e0794d354a45-kube-api-access-pncdx\" (UniqueName: \"kubernetes.io/projected/c1503b13-6950-4c48-bb71-e0794d354a45-kube-api-access-pncdx\") pod \"c1503b13-6950-4c48-bb71-e0794d354a45\" (UID: \"c1503b13-6950-4c48-bb71-e0794d354a45\") " Nov 03 22:30:08.541436 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.541258 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-tmp-dir" pod "c1503b13-6950-4c48-bb71-e0794d354a45" (UID: "c1503b13-6950-4c48-bb71-e0794d354a45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:08.541549 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.541439 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-model-cache" pod "c1503b13-6950-4c48-bb71-e0794d354a45" (UID: "c1503b13-6950-4c48-bb71-e0794d354a45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:08.541549 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.541473 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-home" pod "c1503b13-6950-4c48-bb71-e0794d354a45" (UID: "c1503b13-6950-4c48-bb71-e0794d354a45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:08.541549 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.541539 4963 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-home\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:08.541694 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.541551 4963 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-tmp-dir\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:08.541694 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.541566 4963 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-model-cache\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:08.543644 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.543618 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1503b13-6950-4c48-bb71-e0794d354a45-tls-certs" pod "c1503b13-6950-4c48-bb71-e0794d354a45" (UID: "c1503b13-6950-4c48-bb71-e0794d354a45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:30:08.543644 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.543627 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-dshm" pod "c1503b13-6950-4c48-bb71-e0794d354a45" (UID: "c1503b13-6950-4c48-bb71-e0794d354a45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:30:08.543810 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.543661 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1503b13-6950-4c48-bb71-e0794d354a45-kube-api-access-pncdx" pod "c1503b13-6950-4c48-bb71-e0794d354a45" (UID: "c1503b13-6950-4c48-bb71-e0794d354a45"). InnerVolumeSpecName "kube-api-access-pncdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:08.642460 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.642365 4963 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1503b13-6950-4c48-bb71-e0794d354a45-dshm\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:08.642460 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.642403 4963 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1503b13-6950-4c48-bb71-e0794d354a45-tls-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:08.642460 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.642413 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pncdx\" (UniqueName: \"kubernetes.io/projected/c1503b13-6950-4c48-bb71-e0794d354a45-kube-api-access-pncdx\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:08.655997 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.655963 4963 generic.go:356] "Generic (PLEG): container finished" podID="c1503b13-6950-4c48-bb71-e0794d354a45" containerID="d1eff6c6fdfe79a7e20ce92494ebc65c58ec503658a98fd5d79b1e6ff998ee37" exitCode=0 Nov 03 22:30:08.656173 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.656009 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" Nov 03 22:30:08.656173 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.656029 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" event={"ID":"c1503b13-6950-4c48-bb71-e0794d354a45","Type":"ContainerDied","Data":"d1eff6c6fdfe79a7e20ce92494ebc65c58ec503658a98fd5d79b1e6ff998ee37"} Nov 03 22:30:08.656173 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.656065 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" event={"ID":"c1503b13-6950-4c48-bb71-e0794d354a45","Type":"ContainerDied","Data":"9e2124e51c28f2ea07d7eeb2345b17d7f13d9311fbfce1fd5e0d43b48026eaea"} Nov 03 22:30:08.656173 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.656083 4963 scope.go:122] "RemoveContainer" containerID="d1eff6c6fdfe79a7e20ce92494ebc65c58ec503658a98fd5d79b1e6ff998ee37" Nov 03 22:30:08.665959 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.665932 4963 scope.go:122] "RemoveContainer" containerID="d1eff6c6fdfe79a7e20ce92494ebc65c58ec503658a98fd5d79b1e6ff998ee37" Nov 03 22:30:08.666335 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:30:08.666311 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1eff6c6fdfe79a7e20ce92494ebc65c58ec503658a98fd5d79b1e6ff998ee37\": container with ID starting with d1eff6c6fdfe79a7e20ce92494ebc65c58ec503658a98fd5d79b1e6ff998ee37 not found: ID does not exist" containerID="d1eff6c6fdfe79a7e20ce92494ebc65c58ec503658a98fd5d79b1e6ff998ee37" Nov 03 22:30:08.666397 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.666346 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1eff6c6fdfe79a7e20ce92494ebc65c58ec503658a98fd5d79b1e6ff998ee37"} err="failed to get container status \"d1eff6c6fdfe79a7e20ce92494ebc65c58ec503658a98fd5d79b1e6ff998ee37\": rpc error: code = NotFound desc = could not find container \"d1eff6c6fdfe79a7e20ce92494ebc65c58ec503658a98fd5d79b1e6ff998ee37\": container with ID starting with d1eff6c6fdfe79a7e20ce92494ebc65c58ec503658a98fd5d79b1e6ff998ee37 not found: ID does not exist" Nov 03 22:30:08.812289 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.812242 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j"] Nov 03 22:30:08.816082 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.816055 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j"] Nov 03 22:30:08.966304 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:08.966268 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1503b13-6950-4c48-bb71-e0794d354a45" path="/var/lib/kubelet/pods/c1503b13-6950-4c48-bb71-e0794d354a45/volumes" Nov 03 22:30:09.314032 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:09.313936 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-51b683/simulator-single-main-51b683-kserve-bcd65667c-dvj6j" podUID="c1503b13-6950-4c48-bb71-e0794d354a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.163:8000/ready\": context deadline exceeded" Nov 03 22:30:12.526686 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:30:12.526609 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:30:14.960273 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:14.960237 4963 scope.go:122] "RemoveContainer" containerID="4281c7da6f89292c99b9ebefb6c359ab65b185d15e8b4a2c4b6e2e42f34b7e33" Nov 03 22:30:15.692160 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:15.692127 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" event={"ID":"230144ed-b05d-467a-8ee8-e630bf660cfd","Type":"ContainerStarted","Data":"6a61cdbb2d647ff806fa4cb6cb431ce47265d7e1ee859647d7e1fd5fe6771112"} Nov 03 22:30:16.699544 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:16.699510 4963 generic.go:356] "Generic (PLEG): container finished" podID="230144ed-b05d-467a-8ee8-e630bf660cfd" containerID="6a61cdbb2d647ff806fa4cb6cb431ce47265d7e1ee859647d7e1fd5fe6771112" exitCode=60 Nov 03 22:30:16.700012 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:16.699584 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" event={"ID":"230144ed-b05d-467a-8ee8-e630bf660cfd","Type":"ContainerDied","Data":"6a61cdbb2d647ff806fa4cb6cb431ce47265d7e1ee859647d7e1fd5fe6771112"} Nov 03 22:30:16.700012 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:16.699636 4963 scope.go:122] "RemoveContainer" containerID="4281c7da6f89292c99b9ebefb6c359ab65b185d15e8b4a2c4b6e2e42f34b7e33" Nov 03 22:30:16.700120 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:16.700016 4963 scope.go:122] "RemoveContainer" containerID="6a61cdbb2d647ff806fa4cb6cb431ce47265d7e1ee859647d7e1fd5fe6771112" Nov 03 22:30:16.700310 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:30:16.700284 4963 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895750-bsb2g_odh-ai-gateway-infra(230144ed-b05d-467a-8ee8-e630bf660cfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" podUID="230144ed-b05d-467a-8ee8-e630bf660cfd" Nov 03 22:30:16.817824 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:16.817755 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g"] Nov 03 22:30:17.847842 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:17.847810 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" Nov 03 22:30:17.932552 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:17.932516 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230144ed-b05d-467a-8ee8-e630bf660cfd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230144ed-b05d-467a-8ee8-e630bf660cfd-maas-api-service-ca\") pod \"230144ed-b05d-467a-8ee8-e630bf660cfd\" (UID: \"230144ed-b05d-467a-8ee8-e630bf660cfd\") " Nov 03 22:30:17.932882 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:17.932858 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230144ed-b05d-467a-8ee8-e630bf660cfd-maas-api-service-ca" pod "230144ed-b05d-467a-8ee8-e630bf660cfd" (UID: "230144ed-b05d-467a-8ee8-e630bf660cfd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:18.033839 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:18.033758 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230144ed-b05d-467a-8ee8-e630bf660cfd-kube-api-access-64r7v\" (UniqueName: \"kubernetes.io/projected/230144ed-b05d-467a-8ee8-e630bf660cfd-kube-api-access-64r7v\") pod \"230144ed-b05d-467a-8ee8-e630bf660cfd\" (UID: \"230144ed-b05d-467a-8ee8-e630bf660cfd\") " Nov 03 22:30:18.034479 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:18.034445 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230144ed-b05d-467a-8ee8-e630bf660cfd-maas-api-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:18.036078 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:18.036057 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230144ed-b05d-467a-8ee8-e630bf660cfd-kube-api-access-64r7v" pod "230144ed-b05d-467a-8ee8-e630bf660cfd" (UID: "230144ed-b05d-467a-8ee8-e630bf660cfd"). InnerVolumeSpecName "kube-api-access-64r7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:18.134964 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:18.134891 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64r7v\" (UniqueName: \"kubernetes.io/projected/230144ed-b05d-467a-8ee8-e630bf660cfd-kube-api-access-64r7v\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:18.712848 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:18.712810 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" Nov 03 22:30:18.713082 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:18.712849 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g" event={"ID":"230144ed-b05d-467a-8ee8-e630bf660cfd","Type":"ContainerDied","Data":"a2494eb9f8dbfec8f5fbe89cf4f5d601057f4f7ab34a37489d8d52396545f4f2"} Nov 03 22:30:18.713082 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:18.712926 4963 scope.go:122] "RemoveContainer" containerID="6a61cdbb2d647ff806fa4cb6cb431ce47265d7e1ee859647d7e1fd5fe6771112" Nov 03 22:30:18.868755 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:18.868717 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g"] Nov 03 22:30:18.872915 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:18.872868 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-bsb2g"] Nov 03 22:30:18.966820 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:18.966736 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230144ed-b05d-467a-8ee8-e630bf660cfd" path="/var/lib/kubelet/pods/230144ed-b05d-467a-8ee8-e630bf660cfd/volumes" Nov 03 22:30:29.611877 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:29.611841 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:30.537879 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:30.537799 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:31.741028 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:31.740997 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:32.741625 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:32.741599 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:32.964356 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:32.964325 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:33.890961 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:33.890931 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:34.175292 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:34.175199 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:34.795019 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:34.794987 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:36.335372 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:36.335335 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:36.526553 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:36.526515 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:42.527635 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:30:42.527597 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:30:56.092144 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:30:56.092109 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:12.527660 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:31:12.527619 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:31:42.527847 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:31:42.527756 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:31:56.434810 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:31:56.434775 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:12.527725 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:32:12.527682 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:32:31.801828 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:32:31.801677 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 22:32:31.859525 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:32:31.859498 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 22:32:42.528500 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:32:42.528461 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:32:53.501078 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:32:53.501029 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:32:53.501784 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:32:53.501747 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" podUID="09e130a7-702e-4a97-b02f-ca68a03fa694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:56.778313 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:32:56.778270 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:07.451076 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.451035 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9"] Nov 03 22:33:07.451809 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.451467 4963 cpu_manager.go:414] "containerMap: removing container" podUID="230144ed-b05d-467a-8ee8-e630bf660cfd" containerName="cleanup" Nov 03 22:33:07.451809 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.451479 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230144ed-b05d-467a-8ee8-e630bf660cfd" containerName="cleanup" Nov 03 22:33:07.451809 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.451490 4963 cpu_manager.go:414] "containerMap: removing container" podUID="230144ed-b05d-467a-8ee8-e630bf660cfd" containerName="cleanup" Nov 03 22:33:07.451809 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.451495 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230144ed-b05d-467a-8ee8-e630bf660cfd" containerName="cleanup" Nov 03 22:33:07.451809 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.451503 4963 cpu_manager.go:414] "containerMap: removing container" podUID="c1503b13-6950-4c48-bb71-e0794d354a45" containerName="main" Nov 03 22:33:07.451809 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.451509 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1503b13-6950-4c48-bb71-e0794d354a45" containerName="main" Nov 03 22:33:07.451809 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.451518 4963 cpu_manager.go:414] "containerMap: removing container" podUID="230144ed-b05d-467a-8ee8-e630bf660cfd" containerName="cleanup" Nov 03 22:33:07.451809 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.451524 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230144ed-b05d-467a-8ee8-e630bf660cfd" containerName="cleanup" Nov 03 22:33:07.451809 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.451602 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="230144ed-b05d-467a-8ee8-e630bf660cfd" containerName="cleanup" Nov 03 22:33:07.451809 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.451611 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="230144ed-b05d-467a-8ee8-e630bf660cfd" containerName="cleanup" Nov 03 22:33:07.451809 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.451618 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1503b13-6950-4c48-bb71-e0794d354a45" containerName="main" Nov 03 22:33:07.451809 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.451623 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="230144ed-b05d-467a-8ee8-e630bf660cfd" containerName="cleanup" Nov 03 22:33:07.454963 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.454939 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" Nov 03 22:33:07.552775 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.552737 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:33:07.552984 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.552737 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:33:07.552984 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.552739 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 22:33:07.552984 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.552743 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6whcf\"" Nov 03 22:33:07.552984 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.552747 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 22:33:07.555355 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.555324 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9"] Nov 03 22:33:07.638371 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.638332 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7990612-2ed7-4498-b78b-2892899ae3b4-maas-api-tls\") pod \"maas-api-7c97fc4d98-vp7w9\" (UID: \"b7990612-2ed7-4498-b78b-2892899ae3b4\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" Nov 03 22:33:07.638545 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.638448 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7990612-2ed7-4498-b78b-2892899ae3b4-metrics-certs\") pod \"maas-api-7c97fc4d98-vp7w9\" (UID: \"b7990612-2ed7-4498-b78b-2892899ae3b4\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" Nov 03 22:33:07.638545 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.638496 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7990612-2ed7-4498-b78b-2892899ae3b4-openshift-service-ca\") pod \"maas-api-7c97fc4d98-vp7w9\" (UID: \"b7990612-2ed7-4498-b78b-2892899ae3b4\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" Nov 03 22:33:07.638545 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.638513 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn8l2\" (UniqueName: \"kubernetes.io/projected/b7990612-2ed7-4498-b78b-2892899ae3b4-kube-api-access-sn8l2\") pod \"maas-api-7c97fc4d98-vp7w9\" (UID: \"b7990612-2ed7-4498-b78b-2892899ae3b4\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" Nov 03 22:33:07.739551 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.739458 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7990612-2ed7-4498-b78b-2892899ae3b4-metrics-certs\") pod \"maas-api-7c97fc4d98-vp7w9\" (UID: \"b7990612-2ed7-4498-b78b-2892899ae3b4\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" Nov 03 22:33:07.739551 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.739511 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7990612-2ed7-4498-b78b-2892899ae3b4-openshift-service-ca\") pod \"maas-api-7c97fc4d98-vp7w9\" (UID: \"b7990612-2ed7-4498-b78b-2892899ae3b4\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" Nov 03 22:33:07.739551 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.739528 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn8l2\" (UniqueName: \"kubernetes.io/projected/b7990612-2ed7-4498-b78b-2892899ae3b4-kube-api-access-sn8l2\") pod \"maas-api-7c97fc4d98-vp7w9\" (UID: \"b7990612-2ed7-4498-b78b-2892899ae3b4\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" Nov 03 22:33:07.739886 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.739585 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7990612-2ed7-4498-b78b-2892899ae3b4-maas-api-tls\") pod \"maas-api-7c97fc4d98-vp7w9\" (UID: \"b7990612-2ed7-4498-b78b-2892899ae3b4\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" Nov 03 22:33:07.740184 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.740150 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7990612-2ed7-4498-b78b-2892899ae3b4-openshift-service-ca\") pod \"maas-api-7c97fc4d98-vp7w9\" (UID: \"b7990612-2ed7-4498-b78b-2892899ae3b4\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" Nov 03 22:33:07.742100 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.742082 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7990612-2ed7-4498-b78b-2892899ae3b4-maas-api-tls\") pod \"maas-api-7c97fc4d98-vp7w9\" (UID: \"b7990612-2ed7-4498-b78b-2892899ae3b4\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" Nov 03 22:33:07.742248 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.742225 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7990612-2ed7-4498-b78b-2892899ae3b4-metrics-certs\") pod \"maas-api-7c97fc4d98-vp7w9\" (UID: \"b7990612-2ed7-4498-b78b-2892899ae3b4\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" Nov 03 22:33:07.798042 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:07.798010 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn8l2\" (UniqueName: \"kubernetes.io/projected/b7990612-2ed7-4498-b78b-2892899ae3b4-kube-api-access-sn8l2\") pod \"maas-api-7c97fc4d98-vp7w9\" (UID: \"b7990612-2ed7-4498-b78b-2892899ae3b4\") " pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" Nov 03 22:33:08.076444 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:08.076362 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" Nov 03 22:33:08.450054 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:08.450014 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9"] Nov 03 22:33:08.456347 ip-10-0-130-134 kubenswrapper[4963]: W1103 22:33:08.456315 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7990612_2ed7_4498_b78b_2892899ae3b4.slice/crio-a9e4aca7aba84bd7f501ab5650dc0e8e703f72f881a1bca31f7a2b6f6095ecdc WatchSource:0}: Error finding container a9e4aca7aba84bd7f501ab5650dc0e8e703f72f881a1bca31f7a2b6f6095ecdc: Status 404 returned error can't find the container with id a9e4aca7aba84bd7f501ab5650dc0e8e703f72f881a1bca31f7a2b6f6095ecdc Nov 03 22:33:08.457988 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:08.457965 4963 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:33:08.629528 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:08.629489 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" event={"ID":"b7990612-2ed7-4498-b78b-2892899ae3b4","Type":"ContainerStarted","Data":"a9e4aca7aba84bd7f501ab5650dc0e8e703f72f881a1bca31f7a2b6f6095ecdc"} Nov 03 22:33:09.635538 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:09.635447 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" event={"ID":"b7990612-2ed7-4498-b78b-2892899ae3b4","Type":"ContainerStarted","Data":"c6996327de7e89eb6c78ff34438815d14e1e76816dfc388b8041f5046fe25c64"} Nov 03 22:33:09.635538 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:09.635511 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" Nov 03 22:33:09.748573 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:09.748518 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" podStartSLOduration=1.859422435 podStartE2EDuration="2.748503302s" totalImagesPullingTime="889.080867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:09.747915159 +0000 UTC m=+6338.907987424" watchObservedRunningTime="2026-11-03 22:33:09.748503302 +0000 UTC m=+6338.908575574" Nov 03 22:33:11.049842 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:11.049727 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:11.237951 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:11.237890 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:12.528148 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:33:12.528108 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:33:15.646557 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:15.646525 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" Nov 03 22:33:16.244840 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:16.244802 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn"] Nov 03 22:33:16.249071 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:16.249046 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" Nov 03 22:33:16.304140 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:16.304109 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn"] Nov 03 22:33:16.315853 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:16.315816 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fa867c-a504-485f-bf32-6e578f1b7d5d-maas-api-tls\") pod \"maas-api-7c798d54f8-rdccn\" (UID: \"26fa867c-a504-485f-bf32-6e578f1b7d5d\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" Nov 03 22:33:16.316045 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:16.315946 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26fa867c-a504-485f-bf32-6e578f1b7d5d-metrics-certs\") pod \"maas-api-7c798d54f8-rdccn\" (UID: \"26fa867c-a504-485f-bf32-6e578f1b7d5d\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" Nov 03 22:33:16.316045 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:16.315975 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgtsl\" (UniqueName: \"kubernetes.io/projected/26fa867c-a504-485f-bf32-6e578f1b7d5d-kube-api-access-zgtsl\") pod \"maas-api-7c798d54f8-rdccn\" (UID: \"26fa867c-a504-485f-bf32-6e578f1b7d5d\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" Nov 03 22:33:16.316128 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:16.316043 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fa867c-a504-485f-bf32-6e578f1b7d5d-openshift-service-ca\") pod \"maas-api-7c798d54f8-rdccn\" (UID: \"26fa867c-a504-485f-bf32-6e578f1b7d5d\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" Nov 03 22:33:16.416743 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:16.416701 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fa867c-a504-485f-bf32-6e578f1b7d5d-openshift-service-ca\") pod \"maas-api-7c798d54f8-rdccn\" (UID: \"26fa867c-a504-485f-bf32-6e578f1b7d5d\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" Nov 03 22:33:16.416969 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:16.416770 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fa867c-a504-485f-bf32-6e578f1b7d5d-maas-api-tls\") pod \"maas-api-7c798d54f8-rdccn\" (UID: \"26fa867c-a504-485f-bf32-6e578f1b7d5d\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" Nov 03 22:33:16.416969 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:16.416822 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26fa867c-a504-485f-bf32-6e578f1b7d5d-metrics-certs\") pod \"maas-api-7c798d54f8-rdccn\" (UID: \"26fa867c-a504-485f-bf32-6e578f1b7d5d\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" Nov 03 22:33:16.416969 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:16.416850 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgtsl\" (UniqueName: \"kubernetes.io/projected/26fa867c-a504-485f-bf32-6e578f1b7d5d-kube-api-access-zgtsl\") pod \"maas-api-7c798d54f8-rdccn\" (UID: \"26fa867c-a504-485f-bf32-6e578f1b7d5d\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" Nov 03 22:33:16.417428 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:16.417405 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fa867c-a504-485f-bf32-6e578f1b7d5d-openshift-service-ca\") pod \"maas-api-7c798d54f8-rdccn\" (UID: \"26fa867c-a504-485f-bf32-6e578f1b7d5d\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" Nov 03 22:33:16.419391 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:16.419368 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fa867c-a504-485f-bf32-6e578f1b7d5d-maas-api-tls\") pod \"maas-api-7c798d54f8-rdccn\" (UID: \"26fa867c-a504-485f-bf32-6e578f1b7d5d\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" Nov 03 22:33:16.419502 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:16.419413 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26fa867c-a504-485f-bf32-6e578f1b7d5d-metrics-certs\") pod \"maas-api-7c798d54f8-rdccn\" (UID: \"26fa867c-a504-485f-bf32-6e578f1b7d5d\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" Nov 03 22:33:16.476702 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:16.476666 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgtsl\" (UniqueName: \"kubernetes.io/projected/26fa867c-a504-485f-bf32-6e578f1b7d5d-kube-api-access-zgtsl\") pod \"maas-api-7c798d54f8-rdccn\" (UID: \"26fa867c-a504-485f-bf32-6e578f1b7d5d\") " pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" Nov 03 22:33:16.564440 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:16.564347 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" Nov 03 22:33:16.935103 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:16.935005 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn"] Nov 03 22:33:16.941044 ip-10-0-130-134 kubenswrapper[4963]: W1103 22:33:16.941012 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fa867c_a504_485f_bf32_6e578f1b7d5d.slice/crio-b8417d1ed80fd16c891ff81ff0a64eb7248f0e80879a9045de64742a618dbd0a WatchSource:0}: Error finding container b8417d1ed80fd16c891ff81ff0a64eb7248f0e80879a9045de64742a618dbd0a: Status 404 returned error can't find the container with id b8417d1ed80fd16c891ff81ff0a64eb7248f0e80879a9045de64742a618dbd0a Nov 03 22:33:17.676559 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:17.676515 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" event={"ID":"26fa867c-a504-485f-bf32-6e578f1b7d5d","Type":"ContainerStarted","Data":"b8417d1ed80fd16c891ff81ff0a64eb7248f0e80879a9045de64742a618dbd0a"} Nov 03 22:33:18.683282 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:18.683242 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" event={"ID":"26fa867c-a504-485f-bf32-6e578f1b7d5d","Type":"ContainerStarted","Data":"85961e991fb2d4324056a77ef94d1dc5ddb852ea697b19f01eee9d58aa09c278"} Nov 03 22:33:18.683765 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:18.683341 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" Nov 03 22:33:18.797470 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:18.797404 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" podStartSLOduration=1.898466924 podStartE2EDuration="2.797390334s" totalImagesPullingTime="898.92341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:18.795811837 +0000 UTC m=+6347.955884109" watchObservedRunningTime="2026-11-03 22:33:18.797390334 +0000 UTC m=+6347.957462659" Nov 03 22:33:23.501068 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:23.501025 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:33:23.501747 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:23.501710 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" podUID="09e130a7-702e-4a97-b02f-ca68a03fa694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:24.695795 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:24.695755 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c798d54f8-rdccn" Nov 03 22:33:24.858721 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:24.858689 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9"] Nov 03 22:33:24.858953 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:24.858930 4963 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" podUID="b7990612-2ed7-4498-b78b-2892899ae3b4" containerName="maas-api" containerID="cri-o://c6996327de7e89eb6c78ff34438815d14e1e76816dfc388b8041f5046fe25c64" gracePeriod=30 Nov 03 22:33:25.119637 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.119607 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" Nov 03 22:33:25.232479 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.232229 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7990612-2ed7-4498-b78b-2892899ae3b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7990612-2ed7-4498-b78b-2892899ae3b4-openshift-service-ca\") pod \"b7990612-2ed7-4498-b78b-2892899ae3b4\" (UID: \"b7990612-2ed7-4498-b78b-2892899ae3b4\") " Nov 03 22:33:25.232479 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.232282 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7990612-2ed7-4498-b78b-2892899ae3b4-kube-api-access-sn8l2\" (UniqueName: \"kubernetes.io/projected/b7990612-2ed7-4498-b78b-2892899ae3b4-kube-api-access-sn8l2\") pod \"b7990612-2ed7-4498-b78b-2892899ae3b4\" (UID: \"b7990612-2ed7-4498-b78b-2892899ae3b4\") " Nov 03 22:33:25.232479 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.232319 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7990612-2ed7-4498-b78b-2892899ae3b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7990612-2ed7-4498-b78b-2892899ae3b4-maas-api-tls\") pod \"b7990612-2ed7-4498-b78b-2892899ae3b4\" (UID: \"b7990612-2ed7-4498-b78b-2892899ae3b4\") " Nov 03 22:33:25.232479 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.232349 4963 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7990612-2ed7-4498-b78b-2892899ae3b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7990612-2ed7-4498-b78b-2892899ae3b4-metrics-certs\") pod \"b7990612-2ed7-4498-b78b-2892899ae3b4\" (UID: \"b7990612-2ed7-4498-b78b-2892899ae3b4\") " Nov 03 22:33:25.232988 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.232847 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7990612-2ed7-4498-b78b-2892899ae3b4-openshift-service-ca" pod "b7990612-2ed7-4498-b78b-2892899ae3b4" (UID: "b7990612-2ed7-4498-b78b-2892899ae3b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:25.235231 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.235188 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7990612-2ed7-4498-b78b-2892899ae3b4-kube-api-access-sn8l2" pod "b7990612-2ed7-4498-b78b-2892899ae3b4" (UID: "b7990612-2ed7-4498-b78b-2892899ae3b4"). InnerVolumeSpecName "kube-api-access-sn8l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:25.235359 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.235289 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7990612-2ed7-4498-b78b-2892899ae3b4-maas-api-tls" pod "b7990612-2ed7-4498-b78b-2892899ae3b4" (UID: "b7990612-2ed7-4498-b78b-2892899ae3b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:33:25.235564 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.235537 4963 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7990612-2ed7-4498-b78b-2892899ae3b4-metrics-certs" pod "b7990612-2ed7-4498-b78b-2892899ae3b4" (UID: "b7990612-2ed7-4498-b78b-2892899ae3b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:33:25.333776 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.333731 4963 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7990612-2ed7-4498-b78b-2892899ae3b4-openshift-service-ca\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:25.333776 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.333770 4963 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn8l2\" (UniqueName: \"kubernetes.io/projected/b7990612-2ed7-4498-b78b-2892899ae3b4-kube-api-access-sn8l2\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:25.333776 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.333783 4963 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7990612-2ed7-4498-b78b-2892899ae3b4-maas-api-tls\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:25.334117 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.333793 4963 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7990612-2ed7-4498-b78b-2892899ae3b4-metrics-certs\") on node \"ip-10-0-130-134.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:25.747040 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.746998 4963 generic.go:356] "Generic (PLEG): container finished" podID="b7990612-2ed7-4498-b78b-2892899ae3b4" containerID="c6996327de7e89eb6c78ff34438815d14e1e76816dfc388b8041f5046fe25c64" exitCode=0 Nov 03 22:33:25.747485 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.747063 4963 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" Nov 03 22:33:25.747485 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.747077 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" event={"ID":"b7990612-2ed7-4498-b78b-2892899ae3b4","Type":"ContainerDied","Data":"c6996327de7e89eb6c78ff34438815d14e1e76816dfc388b8041f5046fe25c64"} Nov 03 22:33:25.747485 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.747138 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9" event={"ID":"b7990612-2ed7-4498-b78b-2892899ae3b4","Type":"ContainerDied","Data":"a9e4aca7aba84bd7f501ab5650dc0e8e703f72f881a1bca31f7a2b6f6095ecdc"} Nov 03 22:33:25.747485 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.747170 4963 scope.go:122] "RemoveContainer" containerID="c6996327de7e89eb6c78ff34438815d14e1e76816dfc388b8041f5046fe25c64" Nov 03 22:33:25.758627 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.758597 4963 scope.go:122] "RemoveContainer" containerID="c6996327de7e89eb6c78ff34438815d14e1e76816dfc388b8041f5046fe25c64" Nov 03 22:33:25.759259 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:33:25.759123 4963 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6996327de7e89eb6c78ff34438815d14e1e76816dfc388b8041f5046fe25c64\": container with ID starting with c6996327de7e89eb6c78ff34438815d14e1e76816dfc388b8041f5046fe25c64 not found: ID does not exist" containerID="c6996327de7e89eb6c78ff34438815d14e1e76816dfc388b8041f5046fe25c64" Nov 03 22:33:25.759259 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.759165 4963 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6996327de7e89eb6c78ff34438815d14e1e76816dfc388b8041f5046fe25c64"} err="failed to get container status \"c6996327de7e89eb6c78ff34438815d14e1e76816dfc388b8041f5046fe25c64\": rpc error: code = NotFound desc = could not find container \"c6996327de7e89eb6c78ff34438815d14e1e76816dfc388b8041f5046fe25c64\": container with ID starting with c6996327de7e89eb6c78ff34438815d14e1e76816dfc388b8041f5046fe25c64 not found: ID does not exist" Nov 03 22:33:25.906195 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.906158 4963 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9"] Nov 03 22:33:25.908634 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:25.908593 4963 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c97fc4d98-vp7w9"] Nov 03 22:33:26.966854 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:26.966817 4963 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7990612-2ed7-4498-b78b-2892899ae3b4" path="/var/lib/kubelet/pods/b7990612-2ed7-4498-b78b-2892899ae3b4/volumes" Nov 03 22:33:42.528386 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:33:42.528340 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:33:57.166738 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:33:57.166696 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:12.529891 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:34:12.529846 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:34:38.501941 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:34:38.499222 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:34:38.501941 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:34:38.499289 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" podUID="09e130a7-702e-4a97-b02f-ca68a03fa694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:34:42.534327 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:34:42.534223 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:34:57.416274 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:34:57.416238 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:12.533362 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:35:12.533315 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:35:42.528339 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:35:42.528295 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:35:57.805298 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:35:57.805258 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:12.575123 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:36:12.575038 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:36:23.502078 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:36:23.502027 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:36:23.502739 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:36:23.502702 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" podUID="09e130a7-702e-4a97-b02f-ca68a03fa694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:36:42.527462 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:36:42.527423 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:36:53.501080 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:36:53.501030 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:36:53.501939 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:36:53.501873 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b8kl8" podUID="09e130a7-702e-4a97-b02f-ca68a03fa694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:36:58.149411 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:36:58.149377 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:04.724578 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:37:04.724542 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:12.527635 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:37:12.527596 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:37:31.832070 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:37:31.831946 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 22:37:31.895725 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:37:31.895699 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ccxnp_e3c555a1-e6bd-4042-be58-404cb6693922/console-operator/2.log" Nov 03 22:37:42.527379 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:37:42.527297 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:37:46.276835 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:37:46.276796 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:58.545303 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:37:58.545240 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:12.528736 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:38:12.528694 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:38:40.873085 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:38:40.873044 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:38:40.873600 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:38:40.873115 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-xcb6m" podUID="bbb6adca-8296-435f-96bc-90250221f784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:38:41.076586 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:38:41.076539 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:42.529016 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:38:42.528975 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:38:51.713089 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:38:51.713052 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:58.923801 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:38:58.923768 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:01.212056 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:01.212003 4963 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jggj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:39:01.212606 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:01.212093 4963 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jggj6" podUID="d94d48cc-452b-418a-9d5f-64287cc7b716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:39:12.528954 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:39:12.528848 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:39:24.367385 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:24.367344 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-579685d67f-v4cx7_7e175791-e600-4f34-b7f4-688619a990e1/authorino/0.log" Nov 03 22:39:37.274422 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:37.274387 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6m794_24076c9b-c504-4b0c-82dc-d95b44f0e977/manager/0.log" Nov 03 22:39:38.520696 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:38.520667 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-jww6w_fd4d8972-0ca6-40d6-8bd7-993d611500b4/manager/0.log" Nov 03 22:39:38.809974 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:38.809878 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-gdzc5_760f65bf-694d-407a-888b-c408028109f6/proxy/0.log" Nov 03 22:39:40.632966 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:40.632936 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7c798d54f8-rdccn_26fa867c-a504-485f-bf32-6e578f1b7d5d/maas-api/0.log" Nov 03 22:39:42.527533 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:39:42.527498 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:39:42.935415 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:42.935330 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:46.344195 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:46.344165 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-579685d67f-v4cx7_7e175791-e600-4f34-b7f4-688619a990e1/authorino/0.log" Nov 03 22:39:46.744264 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:46.744230 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-mcs6w_3a26aaaf-9eac-483f-b59e-2fdd595e214c/manager/0.log" Nov 03 22:39:46.983359 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:46.983328 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lcfpk_5359b257-aafe-4a02-b987-786f2b8e3efc/manager/0.log" Nov 03 22:39:49.854932 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:49.854875 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d69f97dbf-99vsk_60769ffb-c98f-4d9e-99b8-e13b5f08f0f6/kube-auth-proxy/0.log" Nov 03 22:39:51.207786 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:51.207757 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-b8kl8_09e130a7-702e-4a97-b02f-ca68a03fa694/istio-proxy/0.log" Nov 03 22:39:51.494752 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:51.494666 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-jggj6_d94d48cc-452b-418a-9d5f-64287cc7b716/istio-proxy/0.log" Nov 03 22:39:52.470375 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:52.470340 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-xcb6m_bbb6adca-8296-435f-96bc-90250221f784/istio-proxy/0.log" Nov 03 22:39:53.042345 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:53.042313 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v_6ec5bfd4-81b3-4a81-a2be-9d832439813f/istio-proxy/0.log" Nov 03 22:39:53.719587 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:53.719556 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7fb544c95-8798s_eb7b7d50-f5bf-45b4-acaa-f1bc19831287/router/0.log" Nov 03 22:39:59.208387 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:59.208350 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:59.395889 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:39:59.395853 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:06.834137 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:06.834101 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:07.024373 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:07.024343 4963 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:12.528408 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:40:12.528355 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:40:16.135432 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:16.135401 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lvbhk_f1eb3e05-8b22-4992-97d9-c71ecc6b99dc/global-pull-secret-syncer/0.log" Nov 03 22:40:17.994585 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:17.994553 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-bd5l5_e863782c-c289-4fa5-93b1-b475358a5001/connection-checker/0.log" Nov 03 22:40:18.434555 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:18.434477 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ghxn6_34ba944d-b672-489d-9acc-fde356202fbe/connection-checker/0.log" Nov 03 22:40:18.872008 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:18.871977 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-kwzxh_5ea4f6f2-4699-46d7-91bc-8391dde191eb/connection-checker/0.log" Nov 03 22:40:19.372567 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:19.372525 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j5mgz_cf6a6e00-44f7-4af0-be4d-3650c552d729/konnectivity-agent/0.log" Nov 03 22:40:19.633401 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:19.633314 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6m794_24076c9b-c504-4b0c-82dc-d95b44f0e977/manager/0.log" Nov 03 22:40:21.131664 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:21.131629 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-134.us-west-2.compute.internal_954da78189a5577b346472d086419753/haproxy/0.log" Nov 03 22:40:21.345442 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:21.345410 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-jww6w_fd4d8972-0ca6-40d6-8bd7-993d611500b4/manager/0.log" Nov 03 22:40:21.467372 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:21.467339 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-gdzc5_760f65bf-694d-407a-888b-c408028109f6/proxy/0.log" Nov 03 22:40:24.954131 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:24.954086 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-579685d67f-v4cx7_7e175791-e600-4f34-b7f4-688619a990e1/authorino/0.log" Nov 03 22:40:25.618749 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:25.618711 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6m794_24076c9b-c504-4b0c-82dc-d95b44f0e977/manager/0.log" Nov 03 22:40:26.366069 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.366042 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-mcs6w_3a26aaaf-9eac-483f-b59e-2fdd595e214c/manager/0.log" Nov 03 22:40:26.757487 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.757446 4963 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw"] Nov 03 22:40:26.757963 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.757943 4963 cpu_manager.go:414] "containerMap: removing container" podUID="b7990612-2ed7-4498-b78b-2892899ae3b4" containerName="maas-api" Nov 03 22:40:26.758040 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.757966 4963 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7990612-2ed7-4498-b78b-2892899ae3b4" containerName="maas-api" Nov 03 22:40:26.758106 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.758093 4963 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7990612-2ed7-4498-b78b-2892899ae3b4" containerName="maas-api" Nov 03 22:40:26.761890 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.761857 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:26.811874 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.811831 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cd72m\"/\"kube-root-ca.crt\"" Nov 03 22:40:26.814528 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.814500 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lcfpk_5359b257-aafe-4a02-b987-786f2b8e3efc/manager/0.log" Nov 03 22:40:26.859142 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.859103 4963 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cd72m\"/\"openshift-service-ca.crt\"" Nov 03 22:40:26.859142 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.859133 4963 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cd72m\"/\"default-dockercfg-77mtt\"" Nov 03 22:40:26.862068 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.862041 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw"] Nov 03 22:40:26.890706 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.890636 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b0c14a8a-56cd-40a4-a337-856cfa256749-podres\") pod \"perf-node-gather-daemonset-tbsqw\" (UID: \"b0c14a8a-56cd-40a4-a337-856cfa256749\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:26.890951 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.890780 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0c14a8a-56cd-40a4-a337-856cfa256749-sys\") pod \"perf-node-gather-daemonset-tbsqw\" (UID: \"b0c14a8a-56cd-40a4-a337-856cfa256749\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:26.890951 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.890885 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llkqg\" (UniqueName: \"kubernetes.io/projected/b0c14a8a-56cd-40a4-a337-856cfa256749-kube-api-access-llkqg\") pod \"perf-node-gather-daemonset-tbsqw\" (UID: \"b0c14a8a-56cd-40a4-a337-856cfa256749\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:26.891098 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.891033 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b0c14a8a-56cd-40a4-a337-856cfa256749-proc\") pod \"perf-node-gather-daemonset-tbsqw\" (UID: \"b0c14a8a-56cd-40a4-a337-856cfa256749\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:26.891158 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.891098 4963 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0c14a8a-56cd-40a4-a337-856cfa256749-lib-modules\") pod \"perf-node-gather-daemonset-tbsqw\" (UID: \"b0c14a8a-56cd-40a4-a337-856cfa256749\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:26.991649 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.991611 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b0c14a8a-56cd-40a4-a337-856cfa256749-proc\") pod \"perf-node-gather-daemonset-tbsqw\" (UID: \"b0c14a8a-56cd-40a4-a337-856cfa256749\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:26.991837 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.991689 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0c14a8a-56cd-40a4-a337-856cfa256749-lib-modules\") pod \"perf-node-gather-daemonset-tbsqw\" (UID: \"b0c14a8a-56cd-40a4-a337-856cfa256749\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:26.991837 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.991718 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b0c14a8a-56cd-40a4-a337-856cfa256749-podres\") pod \"perf-node-gather-daemonset-tbsqw\" (UID: \"b0c14a8a-56cd-40a4-a337-856cfa256749\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:26.991837 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.991740 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b0c14a8a-56cd-40a4-a337-856cfa256749-proc\") pod \"perf-node-gather-daemonset-tbsqw\" (UID: \"b0c14a8a-56cd-40a4-a337-856cfa256749\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:26.991837 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.991750 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0c14a8a-56cd-40a4-a337-856cfa256749-sys\") pod \"perf-node-gather-daemonset-tbsqw\" (UID: \"b0c14a8a-56cd-40a4-a337-856cfa256749\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:26.991837 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.991801 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0c14a8a-56cd-40a4-a337-856cfa256749-sys\") pod \"perf-node-gather-daemonset-tbsqw\" (UID: \"b0c14a8a-56cd-40a4-a337-856cfa256749\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:26.991837 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.991819 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0c14a8a-56cd-40a4-a337-856cfa256749-lib-modules\") pod \"perf-node-gather-daemonset-tbsqw\" (UID: \"b0c14a8a-56cd-40a4-a337-856cfa256749\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:26.992185 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.991834 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b0c14a8a-56cd-40a4-a337-856cfa256749-podres\") pod \"perf-node-gather-daemonset-tbsqw\" (UID: \"b0c14a8a-56cd-40a4-a337-856cfa256749\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:26.992185 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:26.991857 4963 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llkqg\" (UniqueName: \"kubernetes.io/projected/b0c14a8a-56cd-40a4-a337-856cfa256749-kube-api-access-llkqg\") pod \"perf-node-gather-daemonset-tbsqw\" (UID: \"b0c14a8a-56cd-40a4-a337-856cfa256749\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:27.050965 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:27.050861 4963 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llkqg\" (UniqueName: \"kubernetes.io/projected/b0c14a8a-56cd-40a4-a337-856cfa256749-kube-api-access-llkqg\") pod \"perf-node-gather-daemonset-tbsqw\" (UID: \"b0c14a8a-56cd-40a4-a337-856cfa256749\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:27.075854 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:27.075818 4963 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:27.464116 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:27.464007 4963 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw"] Nov 03 22:40:27.472192 ip-10-0-130-134 kubenswrapper[4963]: W1103 22:40:27.472161 4963 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb0c14a8a_56cd_40a4_a337_856cfa256749.slice/crio-26010e6e9854b8940e3481eb2ce9173d50ecbdd43a43714651ea77941d820591 WatchSource:0}: Error finding container 26010e6e9854b8940e3481eb2ce9173d50ecbdd43a43714651ea77941d820591: Status 404 returned error can't find the container with id 26010e6e9854b8940e3481eb2ce9173d50ecbdd43a43714651ea77941d820591 Nov 03 22:40:27.711585 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:27.711550 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-jww6w_fd4d8972-0ca6-40d6-8bd7-993d611500b4/manager/0.log" Nov 03 22:40:28.043252 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:28.043224 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-gdzc5_760f65bf-694d-407a-888b-c408028109f6/proxy/0.log" Nov 03 22:40:28.321136 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:28.321028 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" event={"ID":"b0c14a8a-56cd-40a4-a337-856cfa256749","Type":"ContainerStarted","Data":"6991f56442afa4e48a3051d736c9663feb2d7b794ee85f23c2186dbc4ccddeb9"} Nov 03 22:40:28.321136 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:28.321117 4963 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" event={"ID":"b0c14a8a-56cd-40a4-a337-856cfa256749","Type":"ContainerStarted","Data":"26010e6e9854b8940e3481eb2ce9173d50ecbdd43a43714651ea77941d820591"} Nov 03 22:40:28.321381 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:28.321247 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:28.432751 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:28.432691 4963 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" podStartSLOduration=2.43267663 podStartE2EDuration="2.43267663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:28.431811876 +0000 UTC m=+6777.591884159" watchObservedRunningTime="2026-11-03 22:40:28.43267663 +0000 UTC m=+6777.592748902" Nov 03 22:40:32.089629 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:32.089592 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6m794_24076c9b-c504-4b0c-82dc-d95b44f0e977/manager/0.log" Nov 03 22:40:33.892204 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:33.892170 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-jww6w_fd4d8972-0ca6-40d6-8bd7-993d611500b4/manager/0.log" Nov 03 22:40:34.146964 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:34.146865 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-gdzc5_760f65bf-694d-407a-888b-c408028109f6/proxy/0.log" Nov 03 22:40:34.305824 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:34.305789 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-gkklk_24a97420-efcd-4a6f-b072-1e5792264188/cluster-monitoring-operator/0.log" Nov 03 22:40:34.337058 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:34.337033 4963 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-tbsqw" Nov 03 22:40:39.533589 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:39.533560 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wdvlq_184db55a-c6cf-4e61-aff2-2231a47934f6/node-exporter/0.log" Nov 03 22:40:39.963820 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:39.963788 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wdvlq_184db55a-c6cf-4e61-aff2-2231a47934f6/kube-rbac-proxy/0.log" Nov 03 22:40:40.396307 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:40.396204 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wdvlq_184db55a-c6cf-4e61-aff2-2231a47934f6/init-textfile/0.log" Nov 03 22:40:41.685035 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:41.685005 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d69f97dbf-99vsk_60769ffb-c98f-4d9e-99b8-e13b5f08f0f6/kube-auth-proxy/0.log" Nov 03 22:40:42.527445 ip-10-0-130-134 kubenswrapper[4963]: E1103 22:40:42.527403 4963 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8xh9" containerName="extractor" Nov 03 22:40:42.773815 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:42.773779 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-b8kl8_09e130a7-702e-4a97-b02f-ca68a03fa694/istio-proxy/0.log" Nov 03 22:40:42.986555 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:42.986523 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-jggj6_d94d48cc-452b-418a-9d5f-64287cc7b716/istio-proxy/0.log" Nov 03 22:40:43.835864 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:43.835818 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-xcb6m_bbb6adca-8296-435f-96bc-90250221f784/istio-proxy/0.log" Nov 03 22:40:44.260522 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:44.260495 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-59c5f4c899-vdv2v_6ec5bfd4-81b3-4a81-a2be-9d832439813f/istio-proxy/0.log" Nov 03 22:40:44.896485 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:44.896456 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7fb544c95-8798s_eb7b7d50-f5bf-45b4-acaa-f1bc19831287/router/0.log" Nov 03 22:40:46.057587 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:46.057558 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-rc2h2_1c004a42-94de-4929-8617-d6a95d382361/prometheus-operator-admission-webhook/0.log" Nov 03 22:40:49.902673 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:49.902636 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-579685d67f-v4cx7_7e175791-e600-4f34-b7f4-688619a990e1/authorino/0.log" Nov 03 22:40:50.089491 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:50.089455 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-mcs6w_3a26aaaf-9eac-483f-b59e-2fdd595e214c/manager/0.log" Nov 03 22:40:50.306347 ip-10-0-130-134 kubenswrapper[4963]: I1103 22:40:50.306316 4963 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lcfpk_5359b257-aafe-4a02-b987-786f2b8e3efc/manager/0.log"